My Brain Hurts + Plus More on SWIG

Okay, I guess I can only post one blog a day, so I am adding to this one. Go to end for new stuff.

Between getting my head around callbacks, figuring out where the console actually calls functions, and implementing a corner case feature I am getting really tired of looking at code. Not true, it was fun and I learned a lot. My brain does hurt though...

Ever wonder what happens after:

Con::addCommand()

Not after the actual function call, but when that command you carefully constructed gets called. Ever wonder? If you did not then this blog is probably just noise to you. It is about my trek into the bowls of console command execution and what I did to implement callbacks for a scripting language.

Well if you must know and you are still here then check out the source file compiledEval.cpp inside the console subdirectory. This is where the compiled byte code of the TS scripts gets parsed and the correct functions called. It is a big switch statement for the most part.

Why did I need to dig into this? Because I found a corner case in my callback interface for Python that I could not resolve. If I define a function with a namespace then it is possible for that function to get called like this:

Namespace::Function();

You can call script defined namespaced functions this way as well. So? Well, when a function gets called like that it does not return a SimObject. When there is no SimObject, there is no namespace. I use a lookup table to store the Python function object and it is based upon a namespace lookup result that includes the namespace and function name. If I don't have that information then I cannot lookup the Python function object. Hence, I have no idea what called the callback at this point. Another problem is even if you supply the SimObject as the first parameter all I get is a string. There is no type checking for that object and if I were to supply a namespace as a string by accident it could end up calling an entirely different function if I treat that string as an object lookup. Not pretty at all. Worse yet, is functions defined in TS will call the correct function when called this way. My first inclination was to not allow this corner case. I got very close to that...

Then I did what I did not want to do. I dove into compiledEval.cpp. I found the locations where the Con::addCommand functions are called and hacked a variable into there. I ended up defining a global that will get set to the current namespace right before any Con::addCommand function is called. Right after it sets it to NULL. This allows me to peek into the current namespace and function name of the currently being called function inside of my callback. I don't like it, it is a hack, but at least it is a simple hack. Other methods I have seen were more complicated and that is why I was avoiding it in the first place. So, at least this works. It now behaves like the script defined namespaced functions.

This whole exercise did a few things for me: it showed me how much improvement in the code base there has been since TGE (good job GG!), it taught me a lot about the internals of the console (good thing when attaching foreign languages to the engine), and it helped me debug some problems with my callback code.

And so, if you waded through this blog this long, here is some code that may be useful to you:

// determine if the string is a valid identifier
bool isValidIdentifier(const char *name){
	int slen;
	
	if(!name)
		return false;
	
	slen = dStrlen(name);

	// first letter must be alpha
	if(!dIsalpha(name[0]))
		return false;
	for(int count=1; count<slen; count++){
		if(!dIsalnum(name[count])){
			if(name[count] == '_')
				continue;
			else
				return false;
		}
	}

	return true;
}

I needed this function to reject function names and namespace names provided to my callback export function. I decided to go ultra simple rather than account for all the possibilities of correct function name syntax. For some reason there is no identifier syntax in the T3D docs. That surprised me. I finally found it here. So GG, please add some syntax explanation for a valid function, variable, and namespace identifiers to your official docs. I know us experienced programmers tend to take it for granted, but not every new coder knows that you cannot use a '-' dash or space in there function names.

Anyway, now maybe your brain hurts. :)

BTW, the callback code is complete for generic functions. Now I have that under my belt everything else will be much easier. I now will focus on SimObject transparency, variable change callbacks, transfer of arbitrary binary buffers and text buffers (can you say media streaming?), console text output subscription, and a whole bunch of other "stuff".

What is working now is arbitrary python function exported as a callback for arbitrary TS functions with namespace support, simobjects can be created in python and passed to functions python functions, simobjects can be looked up using name or id from python, simobjects cannot be manipulated yet, and a bunch of other functions. Now that the hard part is done the rest should fall into place. Also, I am developing the interface such that you will be able to add other scripting languages that swig supports. Right now there are some limitations, but as I progress my hope to generalize as much as possible to allow those interfaces to be created using the same DLL. It should even be feasible for them to coexist if the other languages do not require to control the main loop like extending Python does. First and foremost is Python however.

Edit:
I got rid of my hack by doing something more complicated. This is a regular rollercoaster isn't it. I went into Con:: and added a new command called addScriptCommand. I also added a new callback type to handle passing a Namespace object as an additional parameter. Then I added a command to Namespace called addScriptCommand and added a new callback type called extScriptCallback. Finally I added a new handler inside of compiledEval.cpp to check for the new callback type. It calls the new callback with all the other parameters in addition to a pointer to the current Namespace. Now my script based callbacks get the data they need (the whole Namespace object including data on packages!) and there are no hacked global variables. It was actually not that bad to add and the architecture looks like it was designed to be extended like this. Good job GG!

SWIG Again, More on Callbacks:
I know I just blogged on some of this, but I think I am closing in on a good approach for callbacks from multiple languages.

One of my goals is keep my SWIG Python extension from being exclusive to Python. I would like to say add support for Lua as a client or server side scripting interface. Even possibly for user only customizations. I would also like to have bindings for PHP for a server side app. So I am taking my time to think through the process of how do I make this extendable. Today I think I nailed the callback extension support.

One thing that needs to happen is any time I register data with a callback I must have a way to cleanup that data. For instance Python objects need to be unreferenced properly if you cease to use that object. Otherwise you will get memory leaks as the Python garbage collection will not be able to cleanup unused objects. Now if I assume I am using only Python objects this is simple. Whenever I replace a function I get the old object (functions are objects in Python) and Py_XDECREF the object when I no longer use it. This usually only happens when I reregister a function with a different Python function. However, what if the callback was registered for a different language like Lua?

That is what I needed to handle so I needed to make it automagic with the object handling this having no knowledge of what scripting language data it is handling:

// typedef to support callback object
typedef void (*ScriptCBObjectFunction)(void *function, int numparameters, void *parameters);
// callback object for keeping track of scripting language callbacks and data
class ScriptCBObject
{
	public:
		enum CBType
		{
			PythonCallback = 0,
		};	
	private:
		CBType callbacktype;
		void *function;
		// store parameters for use by callback function
		int numparameters;
		void *parameters;
		// a function defined by the scripting language to cleanup stored objects
		ScriptCBObjectFunction cleanupObjects;
	public:
		ScriptCBObject(CBType cbtype, void *func, int numparams=0, void *params=NULL, ScriptCBObjectFunction cfunc=NULL){
			callbacktype = cbtype;
			function = func;
			numparameters = numparams;
			parameters = params;
			cleanupObjects = cfunc;
		}
		~ScriptCBObject(){
			if(cleanupObjects)
				(cleanupObjects)(function,numparameters,parameters);
		}
		CBType getType(){return callbacktype;};
		void* getFunction(){return function;};
		int getNumParameters(){return numparameters;};
		void* getParameters(){return parameters;};
};

Now I create a Python specific callback for handling Python objects:

// cleanup stored python objects
void pyScriptCBObjectFunction(void *function, int numparameters, void *parameters){
	// protect the GIL
	PyGILState_STATE gstate;
	gstate = PyGILState_Ensure();
	
	Py_XDECREF((PyObject *)function);
	Py_XDECREF((PyObject *)parameters);
	
	// protect the GIL
	PyGILState_Release(gstate);
}

Now when I need to cleanup data from a scripting language and allocate new space it is simple:

// grabs the stored callback from whatever framework you use to store callbacks
// I am actually using a "static HashTable<Namespace::Entry*,void*> gScriptCallbackLookup;"
// I just simplified this for clarity.
getcallbackobject((void*&)(tempcbobj));
ScriptCBObject *newcbobj = new ScriptCBObject(ScriptCBObject::PythonCallback, (void *)pyfunc, 0, NULL, pyScriptCBObjectFunction);
if(!newcbobj){
	dSprintf(tempStr, 512, "%s::%s function registraction failed to allocate memory for ScriptCBObject.",nEntry->nameSpace,nEntry->name);
	PyErr_SetString(PyExc_MemoryError, tempStr);
	return NULL;
}	
// stores the callback into whatever framework you use to store callbacks
storecallbackobject((void*)(newcbobj));
// if we got an old callback object delete it, cleanup function will do work of cleaning up objects
if(tempcbobj)
	delete tempcbobj;

You can see how it will clean up the object using the python specific callback. So if this is a Lua function object then it will have a callback specific to Lua. This allows Python to replace a console function that Lua already registered. The python register function will properly invoke the Lua callback to properly dispose of the callback info and any extra data. The same will happen if Lua creates a callback that overwrites the Python callback. So no memory leaks and you get to abstract the interface so it is non-scripting language specific. I am not actually using the "type" information as of now. It will probably go away. Having script language specific callbacks (ScriptCBObjectFunction) kept me from needing to check which type of callback was in the hashtable previously.

I know this code may not be all that clear as it is out of context. When I finally release the code to the Python extension it will make more sense. Mostly this is just me saying, "Look what I did!"