ScriptT3D: Working on a Refactor/Rewrite

ScriptT3D as is works for T3D 1.2 and most likely will work with T3D MIT. However, there are some performance issues I would like to target with this rewrite. SWIG has a mode for producing a less cumbersome intermediate Python class. Meaning there is no Python class in the middle to add to function call latency. There are also some other speedups I can take advantage in this mode. One of those is the function call lookup Python does on an object when it calls an object method. If you use "slots" then you can speedup certain calls which is one reason this type of interface was designed.

I have put on hold some other sub projects in my main project to tackle this interfacing challenge. I really want my code interface to be extremely fast with little overhead in function calls. I also want to tackle something that could revolutionize the interface when it comes to speed.

In my earlier blogs you may have seen some of my posts about PyV8. This is Python wrapper for the V8 Javascript engine. This module is an amazing piece of work. Not only can you run Javascript inside of Python, but each execution context can have its own context. This means user side scripting, internal scripting, server scripting, etc can all run in their own little JIT compiled worlds. However, it does not end there. Go to this page about PyV8 and read about how the interface works in the first paragraph. Let that sink in.

If you didn't get what they were talking about they are saying that they are looking at ways to convert Python to Javascript so it can be compiled. Now, I don't believe that the Advogato PyV8 project is related to the PyV8 project project officially. However the implications are pretty amazing. Now what would be interesting to me is not the ability to compile the Python code in its entirety, but only where it touches the interface to the T3D library. If this can be accomplished then Javascript could literally be interfaced to the engine calls of the T3D engine at compiled code speed. Now, I am a little unsure as to where the official PyV8 project is in that regard, but I can still gain speed benefits even without the compiled Python code.

Here is a sample of PyV8 interfacing code:

import scriptT3D
import PyV8
class js_global(PyV8.JSClass):    
    def exec(self, name, parms):
        scriptT3D.exec(name, parms)
jsg = js_global()
js = PyV8.JSContext(jsg)
js.enter()
...
js.eval("exec("echo","hello T3D from Javascript");")

Obviously this is a very simple and probably erroneous example, but it should illustrate what I am trying to show. The scriptT3D object is a Python library which is imported. I then create an execution context with exposed Python functions. These calls are made available to the Javascript that is run when you perform the js.eval call to evaluate your Javascript code. What does that mean? It means you can very directly provide the T3D API via Python to Javascript and still get the benefits of the Python language as well. If the proper functions are exposed this means that all the current Torque Script code could be converted to Javascript and would then be JIT compiled at runtime. I have converted some Torque Script by hand to Javascript and the conversion is fairly straightforward. I would say it could be done automagically with a well written parser. One of the issues with moving to a different scripting language is "What do I do with my existing codebase?" Well if there was a way to convert to Javascript then it would not be a really big issue except where the code uses TS specific features. Those might have to be converted by hand.

I know there is interest in getting ScriptT3D working with T2D. By taking this route and updating ScriptT3D for the MIT version I am doing part of the necessary to make this available to T2D. The reason this is true is because there are features in the T3D codebase that I rely on. In order to port those over to T2D I have to use the MIT version. There is also the matter of GGs pending update of the T2D code. I am hoping that kicks out soon so that I don't have to write the T2D interface twice. So bear with me if you are waiting for this for T2D.

As of today I have successfully started T3D MIT with basic functions for initializing, running the main loop, and exiting program under Python control. I am using the new "-builtin" option to do this. So far it just seems to "work". My next steps are looking into custom Con::exec calls that not only call the exposed functions from the console, but also remember the call lookup so subsequent calls do not need to do any lookup. I think as long as I stay in the main execution thread this should not be an issue. This was already a limitation on my existing interface so that is not an issue. This combined with the "slots" interface for Python we should get faster calls than the console calls from TS. Then I can give the Javascript context access to that interface and potentially significantly speedup the engines scripting language. There are some callback issues from objects, and Python to TS callbacks can solve those, or even a redirection to Python for those calls would take care of that. Not sure what will be best there.

Edit:
Note:
I am not replacing TS in any way. So one could speed up critical sections of the scripting code without touching the rest of their TS codebase. I am just supposing ways to speedup the entire engine if a TS to Javascript converter were to be written.

Any questions?
Bueller...Bueller...Bueller...