Extending Python with T3D : "Exporting Objects"
by Demolishun · 04/09/2012 (11:22 am) · 2 comments
Alright, what am I talking about exporting Python objects into Torque Script? What is it? Why would you do this?
When extending Python we are generally exporting functions from one codebase into Python compatible calls. This allows the Python code to call the functions from the codebase. Okay, that is simple enough. Now we have a unique codebase called T3D. T3D has a scripting language built into it called Torque Script. Inevitably there are times when it would be useful for the Torque Script to call Python functions. This is what is called a "callback".
The callback allows the library (T3D in this case) to "callback" to the Python code and execute a function. This can be quite useful. For instance what if you have an AIPlayer object and you want to place its "think" function in Python code? You can create a callback that resides in the AIPlayer objects namespace called "think". Now whenever the AIPlayer object calls its "think" function it will call the Python callback.
Now how does exporting a Python object into TS fit into this? This is very similar to exporting a function, but instead we export an object and treat it like a SimObject. You can call the functions defined on the object and access attributes. This would be very useful if you just want to call one function to add say a Python Queue object. Then the TS code can use the Queue object to communicate to whatever the Queue is attached to. In the case of Queues in Python it could be a simple messaging mechanism, another thread, another process external to the main Python process, or even another process on another network. All the while TS thinks it is just talking to a SimObject. This simple interface allows for some beautifully simple abstraction and adds a whole new element of possibly functions to TS. It was also fun to figure out how to write, so I am keeping the functionality.
Now I am going to cover one way NOT to do this.
Originally I thought I could do this:
Sounds great doesn't it? There is one fatal flaw: You cannot override or subclass Namespace. Strictly speaking you can, but the Namespace calls for maintaining Namespaces are pretty much hardcoded to use the Namespace class as is. There are allocs and what not that depend on the Namespace object to always be a Namespace object. So all this work for naught? No, part of my solution does work. I did conceive of a way around this issue, but I was already having to change things I didn't want to. I had to make getNamespace in SimObject and lookup in Namespace virtual functions. One way to make this still work is to redefine the original Namespace lookup function to check for an extScriptObject. Then it could call a different lookup function. That would have worked. However, it just was getting more and more complicated. So I took a step back. Why do I need to override the functionality of the lookup mechanism of Namespace? I don't if I do it another way.
Remember the exporting of functions? Well, I can already export functions in the TS global namespace and in the TS objects namespaces. So why not have the ScriptObject object, when created by the export process, export the functions it knows about at that time?
So now we have a sequence like this:
That is it for now. I need to finish writing the code to access attributes on the exported objects. Thankfully that can all be contained within the extScriptObject except for maybe some virtual function changes in SimObject.
When extending Python we are generally exporting functions from one codebase into Python compatible calls. This allows the Python code to call the functions from the codebase. Okay, that is simple enough. Now we have a unique codebase called T3D. T3D has a scripting language built into it called Torque Script. Inevitably there are times when it would be useful for the Torque Script to call Python functions. This is what is called a "callback".
The callback allows the library (T3D in this case) to "callback" to the Python code and execute a function. This can be quite useful. For instance what if you have an AIPlayer object and you want to place its "think" function in Python code? You can create a callback that resides in the AIPlayer objects namespace called "think". Now whenever the AIPlayer object calls its "think" function it will call the Python callback.
Now how does exporting a Python object into TS fit into this? This is very similar to exporting a function, but instead we export an object and treat it like a SimObject. You can call the functions defined on the object and access attributes. This would be very useful if you just want to call one function to add say a Python Queue object. Then the TS code can use the Queue object to communicate to whatever the Queue is attached to. In the case of Queues in Python it could be a simple messaging mechanism, another thread, another process external to the main Python process, or even another process on another network. All the while TS thinks it is just talking to a SimObject. This simple interface allows for some beautifully simple abstraction and adds a whole new element of possibly functions to TS. It was also fun to figure out how to write, so I am keeping the functionality.
Now I am going to cover one way NOT to do this.
Originally I thought I could do this:
- Create a new object based upon ScriptObject called extScriptObject.
- Associate a Python object to a extScriptObject.
- Override the getNamespace object call in ScriptObject to return a custom Namespace object.
- Override the lookup call in the Namespace object to first check for a Python function call of the same name, if it could not find out, then use the original lookup call.
- Generate a Namespace::Entry on the fly to call a function that would call the Python function on the Python object associated with extScriptObject.
Sounds great doesn't it? There is one fatal flaw: You cannot override or subclass Namespace. Strictly speaking you can, but the Namespace calls for maintaining Namespaces are pretty much hardcoded to use the Namespace class as is. There are allocs and what not that depend on the Namespace object to always be a Namespace object. So all this work for naught? No, part of my solution does work. I did conceive of a way around this issue, but I was already having to change things I didn't want to. I had to make getNamespace in SimObject and lookup in Namespace virtual functions. One way to make this still work is to redefine the original Namespace lookup function to check for an extScriptObject. Then it could call a different lookup function. That would have worked. However, it just was getting more and more complicated. So I took a step back. Why do I need to override the functionality of the lookup mechanism of Namespace? I don't if I do it another way.
Remember the exporting of functions? Well, I can already export functions in the TS global namespace and in the TS objects namespaces. So why not have the ScriptObject object, when created by the export process, export the functions it knows about at that time?
So now we have a sequence like this:
- Create a new object based upon ScriptObject called extScriptObject.
- Associate a Python object to a extScriptObject.
- Export all known callable attributes as functions associated with the extScriptObject namespace.
That is it for now. I need to finish writing the code to access attributes on the exported objects. Thankfully that can all be contained within the extScriptObject except for maybe some virtual function changes in SimObject.
About the author
I love programming, I love programming things that go click, whirr, boom. For organized T3D Links visit: http://demolishun.com/?page_id=67
#2
I do know one performance gain we will both get. The ability to integrate mature codebases supported by other people. The gain will be in the time it takes for us to integrate the code into the T3D codebase. ie it will make us much more efficient writing new code. That was part of the motivation for doing the Python connection in the first place. I get more done, and the code is higher quality in Python than in C++. No pointers, no obscure errors, and no seg faults, and much more robust code.
At some point I will assess the use of decorators. I know it will help cleanup the interface to a large degree. Another item I need to address is Python code and TS code that does some form of unit testing (not sure that is the right term). I want to have some code I run that will stress the various interfaces and assert conditions to make sure any changes I make don't affect something else. This will be a big part of making sure everything is robust. I hate flaky code.
The callbacks are only elegant because I want to put myself through more pain. I am holding on to the delusion that I will "want" to add support for other scripting languages. If I did not have that fantasy I might be done by now. Part of that mental disease makes me think people will pick up what I started and hack away until they add support for the other scripting languages SWIG supports. God knows that once I get the code mature and stable I am totally a user and minor maintainer at that point. It hurts just thinking about the brain damage. Have you seen my crowbar?
Thanks for dropping by. I always enjoy our discussions.
04/10/2012 (11:40 pm)
I am excited to see the performance gains on your implementation. I suspect, based upon your experience, to get some performance gain only because Python is more mature than TS. Then again, who knows. Until I get hard numbers it is pretty moot. I do know one performance gain we will both get. The ability to integrate mature codebases supported by other people. The gain will be in the time it takes for us to integrate the code into the T3D codebase. ie it will make us much more efficient writing new code. That was part of the motivation for doing the Python connection in the first place. I get more done, and the code is higher quality in Python than in C++. No pointers, no obscure errors, and no seg faults, and much more robust code.
At some point I will assess the use of decorators. I know it will help cleanup the interface to a large degree. Another item I need to address is Python code and TS code that does some form of unit testing (not sure that is the right term). I want to have some code I run that will stress the various interfaces and assert conditions to make sure any changes I make don't affect something else. This will be a big part of making sure everything is robust. I hate flaky code.
The callbacks are only elegant because I want to put myself through more pain. I am holding on to the delusion that I will "want" to add support for other scripting languages. If I did not have that fantasy I might be done by now. Part of that mental disease makes me think people will pick up what I started and hack away until they add support for the other scripting languages SWIG supports. God knows that once I get the code mature and stable I am totally a user and minor maintainer at that point. It hurts just thinking about the brain damage. Have you seen my crowbar?
Thanks for dropping by. I always enjoy our discussions.
Torque Owner Vince Gee
WinterLeaf Entertainment
It's funny how we took two different approaches to the problem of callbacks. I must say, I think yours is more elegant than mine.
I simply wrote a c++ engine console to handle calling back to the csharp. On application startup, I seed the console with torquescript functions that wrap the callback, so I end up with something like this being injected into the torquescript
This is all generated dynamically via csharp reflections, so the programmer only has to put a decoration before the csharp function they want to export and it would generate something like the following code.
function AIPLayer::Think(param1,param2, etc, adds params based on what the csharp expects.) { return CallCsharp(SomeFunctionName,param1,param2,... etc); }I feel the biggest difference in our designs is that my design is more about "Remoting" the Torque Dll, where yours is more like integration.
Both achieve the same goals, I'll be curious to see how each performs.
Right now, I'm up to something like 3000 externs being exposed, I finally managed to figure out a method to parse the old style console functions and methods. So now, pretty much all old style console methods and new style console methods are exposed as base type externs in my code base. I really noticed a significant uptick in performance once I was able to get rid of 90 percent of calls to the console and instead modify the objects inside of torque directly through c++ externs.
Unfortunately, there are still about 200-300 console calls and such that just couldn't be converted to externs due to their complexity. For those guys I'll need to hand convert them.
Vince