scriptT3D extension for Python

Requirements:
Windows OS (Windows XP and above I would assume)
Python 2.7 (32 bit)
T3D 1.2 (It may work for previous versions, but is untested by myself. Entr0py got it working with 1.1.)
T3D MIT 1.2 for MIT Release (Most likely would work fine with 1.2/1.1)
SWIG 2.0.4 (if you are rebuilding source from the .i files)
SWIG 2.0.8 for MIT Release (2.0.4 may work, but i upgraded to 2.0.8)

Code:
MIT ScriptT3D 1.2
Source files are included for engine source files. So you won't need to manually change engine files unless you have a highly modified version of T3D.

older scriptT3D 1.1
older scriptT3D 1.0
The older versions contain a diff against the engine source, the source files after generation through

SWIG, the SWIG files in case you want to modify the interface, the generated Python code, and sample Python scripts.

MIT Version of ScriptT3D

MIT ScriptT3D 1.2 is the version of ScriptT3D that works with the MIT 1.2 version of T3D. The ScriptT3D source code is officially released under the MIT license as well. This is in keeping with the desire to create a mind share of code development, but not limit commercial use of the code. Most of the documentation below still applies. However, the API is radically different. There is limited example code as well as unit tests. In addition there are some text files that help explain some parts of the interface. My intention over the next few months will be to convert the old examples, add new examples, and add new features to this interface. I am ramping up my development efforts on a particular project and this codebase will be the foundation for that work. So when I develop particularly useful features, fix bugs, and otherwise abuse this code those changes will end up here.

Description:
There is lots of documentation in the Python scripts on how to use the interface. The difficult part is how to structure your source code. I put everything including the modified engine source under the source directory in the project directory. This allows me to keep the source files separate from the engine source files. This includes the engine source files I changed. There is also instructions on how to setup SWIG to work with VC++. Look through the source files as I tried to document the interface in the comments where ever I could. This will hopefully make it so I can remember what I did 6 months from now.

Here is a link to a resource that shows how to structure your source in your project: www.garagegames.com/community/resources/view/21664

I included the directory structure in the zip file of how the sub-directories should look. Of course if you want to modify the engine source in the normal engine source directory that is up to you.

The diff file contains all the changes needed to the engine source files. Just ignore the lines that say "Only In". Also, use Wordpad to view the files if running under Windows. Notepad just does not handle *nix style files very well. I used Gnu DiffUtils for Windows to produce the diff.

Please have a look and see what needs to be improved not only for the actual code, but for the installation instructions. I have been getting burned out looking at the code and would like to get it into your hands now. I very much intend for you to add, change, improve, etc. If something is hard to setup or use then let me know and I will tweak this resource to help make it easier for the next person.

Have fun!!!

Install Instructions:

1. Create a new project or use existing one.
2. Copy the zip_source.zip file to your project source directory. Usually located in the C:/Torque/Torque 3D 1.2/My Projects/<project name>/source
3. Unzip the contents into the source directory.
4. Run the getEngineSourceFiles.bat to copy the needed engine sources to be modified.
5. Look at the changes in changes_diff.txt to modify the source files you just copied into the project source directories.
6. Add the following project source files to your projects DLL project in VC++: scriptT3D.cpp, scriptT3D.h, scriptT3D_wrap.cxx, app/mainLoop.cpp, console/compiledEval.cpp, console/console.h, console/consoleInternal.cpp, console/consoleInternal.h, console/scriptObjects.h, console/simObject.h
7. Add the the following project sources files to your projects executable project in VC++: main/main.cpp.
8. Set the DLL project to produce: _scriptT3D.pyd instead of the <projectname>.dll file.
9. Everything should now compile correctly.
10. The main app should run be compiled and will run as well because we modified it to use the _scriptT3D.pyd library as well. BTW, pyds are still DLLs. So they can be used like DLLs under Windows.
11. Copy scriptT3D.py to your projects "game" directory. Copy the files in the pyscripts directory to the "game" directory.
12. Now run the test scripts and read through them to learn how to use the code.

Python Interface Theory of Operation:
I realized there is not a good description of how this works at the function call level. There is some confusion to this effect. Here I will try and describe what takes place for the various interfaces and function calls. This may only be useful to those really needing to know how it works under the hood.
Notes:
TS means Torque Script.

1. SimObject creation: For SimObject creation (and derived objects) I rely on the console to do the heavy lifting. The reason is there is a lot going on when creating an object and I figure it is best for the existing console code to handle this. This is generally handled through the use the Evaluate command. This is usually called like this scriptT3D.Sim().Evaluate(<string to evaluate>).
2. Python objects are handled in Python of course. However, they can be imported into the T3D and will have a SimObject host. This SimObject will call methods that are detected at the time of import as executable. It will also access attributes of various types natively in TS. That means you can literally import a Python object and access it in TS as if it were a TS object. Calling a function on an import Python object directly calls the method on the corresponding Python object. It is very fast. Note: For the 1.2 MIT version this option was removed. I did not feel good about the code. If people request I can put this back in. I have not received a lot of feedback on this particular feature so I don't think it will be missed. You can always simulate this feature using callback exports anyway.
3. Calling functions on a TS object from Python is fast, however it does require a lookup in Python to verify the attribute is actually a function, and then it calls the function directly. The only time it does not call the function directly is if the :: method notation is used. This is not a typical way to call a function. In that case it will use Evaluate. It is also not a recommended way to call a function and is only supported for legacy reasons. It is also not how you call functions in Python. You have to use special syntax to call a function that way.
4. Callbacks are a way to call Python functions directly from TS. You can create global callbacks in TS that will call functions in Python. In fact, when you import a Python object into TS it creates callbacks for every attribute that is detected as a method or function. This allows for TS objects as part of their normal processing to call Python functions without doing anything special. For instance, you could create a 'think' callback against the AIPlayer namespace. Then every time an AI 'think' method is invoked it would actually call the Python method associated with the callback. Callbacks provide a very powerful way to invoke Python code as if it were part of the TS code.

Hopefully this explains how the functions are actually invoked in scriptT3D for the Python interface. It is as close to the metal as I could get it at this point. SWIG, the interface library that helps me create this interface is transitioning to more direct calls rather than using an intermediate Python object. I tried using this method and it failed to work properly. Once that gets more stable I may be able to get the calls be a little bit faster than they are now.

Python Issues:
You may need to tweak Python for this to work properly. The issue arises from Python development libraries using a debug symbol and requiring a separate debug library for development. I don't recommend using this additional library/libraries. This can be fixed by tweaking a header file in the Python headers. Look in C:/Python27/include/pyconfig.h. Change the lines that look for the _DEBUG symbol being present. Change the line that try to require the debug version of the Python lib to look like this:

ifdef _DEBUG
#				pragma comment(lib,"python27.lib")
#			else
#				pragma comment(lib,"python27.lib")
#			endif /* _DEBUG */

Also disable the Py_DEBUG symbol by commenting it out like this (notice the slashes):

#ifdef _DEBUG
//#	define Py_DEBUG
#endif

Changes:
5/29/2012: Changed how global variables and simobjects are accessed through Sim(). Example:

print Sim().o.objectname
Sim().o.objectname = object # illegal, will fail
print Sim().g.globalname
Sim().g.globalname = "blah, blah, blah"

9/28/2012: Fixed example for new API:

import sys
import scriptT3D
import sched, time
import Queue

# use for scheduling events
#   This allows us to have functions occur after a delay.
#   Good for doing tests that are created before the main loop, but execute in the main loop.
#   This may or may not be used in this program
s = sched.scheduler(time.time, time.sleep)

# just assigning the object to a variable
#   Just a preference
engine = scriptT3D
# Each time you run engine.Sim() it creates an object to access the console.
#   Not a big deal, but just to help you be aware of that.  I had issues trying
#   to create a singleton.
console = engine.Sim()
conObjects = console.o
conGlobals = console.g

# put everything in a giant exception check for testing
#   Not recommended for normal functioning.
#   I got tired of everything freezing before the engine loop...
try:

    # init
    #   Sends command line args to engine
    if not engine.init(len(sys.argv),sys.argv):
        sys.exit(1)

    # console consumer callback
    #   Sends console output to Python stdout.
    def consumerCallback(level, line):
        # notice you can tell the level # (normal, warning, error) of the message
        print level, line

    # export consumer
    engine.ExportConsumer(consumerCallback)

    # execute our own startup
    #   For my setup I rename game.cs (which is called by default) in the root directory to main_hook.cs.
    #   I create an empty game.cs so that the engine does not complain.
    #   If you prefer to use the game.cs as is then this should just give an error in the console output.
    #   My intent is to be able to more finely control the startup sequence.  Eventually I will have
    #   Python be in charge of more of the startup like creating objects and setting up the canvas.
    #   This will take some time to figure out, but in the meantime we can use the normal TS startup.
    #   Another intent is to allow pieces of the engine to be tested out.  This gives you the ability
    #   to create objects and test their interfaces without the rest of the engine getting in the way.

    # create main hook
    def main_hook():
        print "executing main_hook()"
        console.Evaluate(""" exec("main_hook.cs"); """)

    # execute the main_hook
    main_hook();

    # export a function
    def exfunction(data):
        print data
        return len(data)
    # exported into global namespace
    engine.ExportCallback(exfunction,"exfunction")
    # call using evaluate
    print console.Evaluate("""exfunction("hello");""")
    # call using execute
    #   Takes a SimObject (name or id), length of the argument list, and an argument list.
    #   If no SimObject, ie a global function, then provide empty string "".
    #   The first argument is the function name.
    #   It may make sense to make this simpler.
    arglist = ["exfunction","world"]
    print console.Execute("",len(arglist),arglist)

    # messing with global data
    console.g.global_variable = "Some Data"
    print console.g.global_variable
    console.g.another_global = "Some more data..."

    # creating a SimObject and get the id as a string
    #   It is best to let T3D create its own objects.  Hence the use of the evaluate function
    simobj = console.Evaluate("""%temp = new SimObject(UniqueObjectName){attrib1="some value";}; return %temp;""")
    print simobj # prints the id
    # use id to get a reference to a python SimObject
    psimobj = console.FindObject(simobj)
    # now print out attrib1
    print psimobj.attrib1
    # another way, if you know the object name or if you have the number, problem is you cannot put the number in from a variable easily
    print console.o.UniqueObjectName.attrib1
    # can set it this way too
    console.o.UniqueObjectName.attrib1 = "some other value"
    # make more attributes
    console.o.UniqueObjectName.attrib2 = "can also make new attributes"
    # get the object Torque Script equivalent
    #   Can be a way to save object states to a database...
    print repr(psimobj)

    # export a queue object
    q = Queue.Queue()
    # returns the extScriptObject created to hold the exported object
    #   Mainly useful for using in TS functions
    qobj = engine.ExportObject(q)
    # this just shows the object type
    #   Notice the unique name generated by the export.
    #   This keeps the namespace unique for each object.
    print repr(qobj)
    # get the id for use in evaluate scripts
    print qobj.getID()
    # show the functions and variables that got exported from the queue object
    #   This calls the 'dump' function of the SimObject.
    #   There is a lot of junk in the 'dump', but you can see the methods that got exported like: put, qsize, get
    qobj.dump()
    # testing putting something on the queue
    console.Evaluate("""{0!s}.put("{1!s}");""".format(qobj.getID(),'a message on the queue'))
    # now test pulling from the queue
    print "queue message:",q.get_nowait()

    # function callbacks
    #   These are very interesting because I had to make some decisions to get
    #   the ability to use any function as a callback, but still have the ability
    #   to detect which SimObject, if any, it was called against.  If I had not
    #   made this decision then it would be harder to export entire objects unless
    #   I used a different callback mechanism.  I finally decided to use one solid
    #   mechanism, but for every callback you can still see the SimObject if any
    #   the function was called against.  This is possible because functions are
    #   objects themselves and are capable of having attributes set on them.

    # lets create a function to test this on
    def test_callback():
        if(hasattr(test_callback,"__SimObject__")):
            print "Called from the namespace of this object:",test_callback.__SimObject__,test_callback.__SimObject__.getName()
        else:
            print "Not called from SimObject namespace"

    # now lets export some callbacks
    #   callback export syntax: ExportCallback(<function>,"name","namespace"=None,"usage"=None,override=True)
    #   function is the function or method
    #   name is the function name in TS
    #   namespace is the namespace, can be None if not used
    #   usage is the usage string, can be None if not used
    #   override is if this will override existing functions, default is True, if false it will kick back an error if override is attempted
    #   Every parameter after name is optional.
    engine.ExportCallback(test_callback,"test_callback") # global namespace
    engine.ExportCallback(test_callback,"test_callback","SimObject") # SimObject namespace
    console.Evaluate("""test_callback();""") # global call
    console.Evaluate("""{0!s}.test_callback();""".format(simobj)) # test against simobject we created earlier

    # Callbacks against class methods are the same as above.  Python class callbacks already
    #   know which Python class they are called on.  Just use the __SimObject__ attribute to find
    #   out which SimObject called the callback if any.

    # some more info on object exporting
    # complex class
    class cattrib(object):
        def __init__(self):
            self.attr1 = ['a','b']
            self.attr2 = (0,1)
            self.attr3 = {'one':1,'two':2}
            self.attr4 = int(5)
            self.attr5 = float(1.2)
            self.attr6 = str('doodle')
    # export an instance of the cattrib object
    cobj = engine.ExportObject(cattrib())
    # use id to call functions in TS
    # accessing arrays, tuples, dictionaries, stings, and numbers are supported
    #   When setting attributes from TS the functions try and maintain data if
    #   the attribute already exists.  If not it will default to a type like string
    #   or dictionary as they are easist to convert to a Python representation.
    #   So if you want a particular datatype then define it on the Python side first.
    #   Also note that if the key or index for a sequence (array, tuple, dictionary)
    #   is of the wrong type it will fail to set or read the sequence.  Tuples and strings
    #   are not able to be written as they are immutable types.
    console.Evaluate("""echo({0!s}.attr1[1]);""".format(cobj.getId()))
    console.Evaluate("""echo({0!s}.attr2[0]);""".format(cobj.getId()))
    console.Evaluate("""echo({0!s}.attr3["two"]);""".format(cobj.getId()))
    console.Evaluate("""echo({0!s}.attr4);""".format(cobj.getId()))
    console.Evaluate("""echo({0!s}.attr5);""".format(cobj.getId()))
    console.Evaluate("""echo({0!s}.attr6);""".format(cobj.getId()))

    # hwnd handle
    #   This is so other apps can use the T3D hwnd handle.  I have tested this with
    #   wxPython and it does work.  However, wxPython event system interferes with the
    #   T3D event system so it is not recommended to be used with wxPython.  Other
    #   packages I would try to use this with is pyOgre and pyGame.  Those have independent
    #   event systems that are not needed.  Now, these graphicssystems are not necessarily
    #   any better than T3D, it is I just wanted to have the capability.  There are other
    #   GUI systems that might make sense to use with T3D.
    hwnd = engine.gethwnd()

    # run the main loop
    while engine.tick():
        # run any scheduled events
        s.run()

    # normal shutdown
    engine.shutdown()

except Exception,e:
    print e

Contributed:
Look at the posts made by Guy Allard to this resource. He has created some really useful code templates for testing out the engine interface.

Credits:
Thanks to everyone in this community for just being your awesome selves.
Thanks to:
GarageGames for making an awesome engine.
Python developers, of which there are too many to count.
Vince Gee for giving me really good ideas on where to take this interface.
Guy Allard for being the guinea pig.
My Creator, for giving me the drive to create.