Extending Python with T3D

Python is a very powerful development environment. It allows the programmer to have access to some of the highest quality libraries available on the internet. Thousands of libraries have been used to extend Python to make it the ultimate 'glue' language. It can be used in a web server, the desktop, to control super computers, and many other tasks.

T3D is a very powerful and complete game engine. However, every time there is a need for a library that is not part of the Torque code base it must be linked into the Torque engine. Often times this process requires turning the library into console object functions and data. This can be time consuming and error prone. Python has a unique advantage of getting rid of this scenario. Many people have already spent the time to integrate and debug connecting an app or library to Python. So if T3D can be made to connect with Python we get instant access to the wealth of stable and tested Python libraries. This is where this tutorial comes in. It describes how to 'extend' Python to make T3D act as a library for Python with full console object access.

The Good News
Someone has already taken the time to integrate Python with T3D. Actually they did it with TGE. This was done by the folks at [http://www.mmoworkshop.com/trac/mom/wiki/PyTorque Prairie Games]. They have a working MMO that is very good. The amount of blood and sweat that has gone into the MoM MMO is astonishing. If you have not played it check it out [http://www.minionsofmirth.com/ here].

The Bad News
Out of the box the code available [http://www.mmoworkshop.com/trac/mom/wiki/PyTorque here] does not work with T3D, it works with TGE. But that is what this article is all about...which leads us to...

The Great News
It did not take long to determine what changes needed to be made to extend Python with T3D. The T3D codebase is actually more suited to extending Python than the TGE codebase. The reason is because most of the T3D engine is already a library! This allows the new functionality to be logically split from the original code. It also allows us to add a few more functions to the DLL to allow it to talk to Python.

The Fine Print
Extending Python with T3D is not going to make the engine suddenly independent of Torque Script. You still have to have a main.cs and it has to have a minimum of certain modules loaded. The reason is the main.cs must create a canvas at some point in the Initialization of the engine. There is still a significant amount of work that needs to go into using Python with T3D. It has no integration to the built in editors and has no knowledge of the asset directory structure. This is very much a barebones setup and will require a significant amount of assembly. My goal with this article is to invite people to build their Python toolsets and share some of that information with the GG community. I thought about putting this information later in the article. However I wanted to warn the reader that there is much more to do after this extension is completed.

Reference Information

Links
Garage Games
Prairie Games
Python
Extending Python
TDN PyTorque

Source Code
PyTorque
pytorque.cc diff

Credits

Garage Games
For creating the engine and making this possible.

Prairie Games
For doing the hard work of figuring out how to make Torque objects play nice with Python objects.
If you have not played their game [http://www.minionsofmirth.com/ Minions of Mirth] I highly recommend it. The game is very inspiring as to what can be achieved with the TGE and/or T3D engines.

Nuts and Bolts

Version Note
This article was written using T3D 1.1 and 1.2, Python 2.6, and PG's PyTorque code.
I am sure 2.7 of Python will work fine, but if you run into issues go back to 2.6 to remove that as a factor. Once you have it working experiment with 2.7. I may be surprised, but I do not think this will work with Python 3.x. Just as a note of interest the latest version of MoM (as of this writing) uses 2.7.

Licensing Note
I cannot provide files with the modifications made in this resource. Do NOT upload them into this resource. The reason is the PyTorque code license if very similar to the Torque code license. The license does not allow for hosting the code on another website for public consumption. PyTorque code was developed by PG and is owned by PG. Respect their license by not hosting the code somewhere else. The most I can provide is diff files and links to the PyTorque source code. The reason PG provides the PyTorque source code is to draw traffic to their products. By hosting the code somewhere else you are stealing their traffic. They have been very gracious to provide this code to the community. To ensure the PyTorque resource is maintained please visit their site and purchase their products.

Create a New Project
If you don't already know how to create a project follow this link and learn how to do that: Your First Project Tutorial. Make sure you create a "full" project for this.

Get the PyTorque Source Code from PG
The source code is located here: PyTorque. Click on the PyTorque in the side menu and get the file from the Downloads section.

Backup the Engine Source Code
The engine source is typically located here: "C:TorqueTorque 3D Pro 1.1Engine" on a Windows system. Make a copy of the 'source' directory or backup the 'source' directory to another location. I just made a backup by doing a copy and paste. That way I ended up with a directory named 'source - Copy'. The reason for this step is that this part of the source code is shared across projects. We will be making changes to this code and it is always nice to be able to compare against an untouched file to see what was changed.
Of course, if you are comfortable making these changes or already have some sort of revision control setup then feel free to skip this step.

Copy PyTorque Code Into the Engine Directories
This step is to place the PyTorque code into the engine directories in the correct locations.
If your base directory for your Torque install differs from what is shown here just make the necessary substitution.
1 Unzip the pytorque.zip file somewhere.
2 From the 'tge15enginecore' directory copy tDictionary.h and tDictionary.cc to the 'C:TorqueTorque 3D Pro 1.1Enginesourcecore' directory.
3 Create a new directory under the 'C:TorqueTorque 3D Pro 1.1Enginesource' directory and call it 'python'.
4 From the 'tge15enginepython' directory copy pytorque.cc to the 'python' directory you just created.

Add Python Development Headers and Libraries to Visual C++ Project
1 Open up the Visual C++ project you created from the Torque Toolbox.
2 Goto the 'DLL' project and select 'Properties'. Update the C/C++ 'Additional Include Directories' to include 'C:Python26include' directory.
3 Do the same for the Linker 'Additional Library Directories' to include the 'C:Python26libs' directory.

Adjust Project to Produce Python Library
In the DLL Project go to Properties->Linker settings again and change the 'Output File' from the dll to 'pytorque.pyd'. If you want this can be name anything as long as the extension is pyd. Also, if you want to be able to run the non-dll portion of the project you created you can change the dll opened in main.cpp as follows:

//sprintf(gameLib, "%s.dll", filename);
sprintf(gameLib,"pytorque.pyd");

Adding files to the project
Add tDictionary.h, tDictionary.cc, and pytorque.cc to the 'DLL' project. At this point the project will fail compilation if you try to compile.

Changes to pytorque.cc
1 Comment out this include: #include "platform/gameInterface.h"
2 At the end of the file comment out the functions: Torque_Initialize, Torque_Tick, and Torque_Shutdown.
3 Add this code at the end of the file:

extern "C"
{
	bool torque_engineinit(int argc, const char **argv);
	int  torque_enginetick();
	bool torque_engineshutdown();
};

static int Torque_Initialize(int argc, const char** argv)
{
	if (!torque_engineinit(argc, argv))
		return 1;

	gPyTorqueInitialized = true;

	return 0;
}

static int Torque_Tick()
{
	return torque_enginetick();
}

static int Torque_Shutdown()
{
	torque_engineshutdown();

	gPyTorqueInitialized = false;

	return 0;
}

4 In the function 'cTorqueToPythonCallback' (line ~231) change this:

//why is argc one too many? only in the case of object?
argc-=1;

to this:

//why is argc one too many? only in the case of object?
//argc-=1;

5 In the function 'pytorque_export' (line ~393) change this:

gCallableLookup.insertEqual(lookup,callable);


   Con::addCommand(ns,fname,cTorqueToPythonCallback,usage,minargs+1,maxargs+1);
}
else

to this:

gCallableLookup.insertEqual(lookup,callable);


   Con::addCommand(ns,fname,cTorqueToPythonCallback,usage,minargs,maxargs);
}

6 In the function 'pytorque_export' (line ~410) change this:

}
   
   Py_INCREF(Py_None);
   return Py_None;

to this:

}

   // must increment ref for stored object references
   Py_XINCREF(callable);
   
   Py_INCREF(Py_None);
   return Py_None;

Changes to compiledEval.cpp
1 Below line 540 (1.1)/ 537 (1.2) change this:

U32 lineNumber = code[ ip + 3 ] >> 1;

#ifdef TORQUE_DEMO

to this:

U32 lineNumber = code[ ip + 3 ] >> 1;
// added for PyTorque
CodeBlock::sfnNameSpace=fnNamespace;

#ifdef TORQUE_DEMO

2 Below line 1481 (1.1) / 1489 (1.2) change this:

fnName      = U32toSTE(code[ip]);

// Try to look it up.

to this:

fnName      = U32toSTE(code[ip]);
// added for PyTorque
CodeBlock::sfnNameSpace=fnNamespace;

// Try to look it up.

3 Below line 1638 (1.1) / 1646 (1.2) change this:

STR.popFrame();
}
else
{
   switch(nsEntry->mType)

to this:

STR.popFrame();
}
else
{
   // added for PyTorque
   CodeBlock::sfnNameSpace=nsEntry->mNamespace->mName;

   switch(nsEntry->mType)

4 Below line 1940 (1.1) / 1949 (1.2) change this:

execFinished:
   if ( telDebuggerOn && setFrame < 0 )

to this:

execFinished:
   // added for PyTorque
   CodeBlock::sfnNameSpace=NULL;

   if ( telDebuggerOn && setFrame < 0 )

Changes to codeBlock.cpp
1 Below line 22 change this:

ConsoleParser *CodeBlock::smCurrentParser = NULL;

//-------------------------------------------------------------------------

to this:

ConsoleParser *CodeBlock::smCurrentParser = NULL;
// added for PyTorque
StringTableEntry CodeBlock::sfnNameSpace=NULL;

//-------------------------------------------------------------------------

Changes to codeBlock.h
1 Below line 26 change this:

static Compiler::ConsoleParser * smCurrentParser;

static CodeBlock* getCurrentBlock()

to this:

static Compiler::ConsoleParser * smCurrentParser;
// added for PyTorque
static StringTableEntry CodeBlock::sfnNameSpace;

static CodeBlock* getCurrentBlock()

Changes to mainLoop.cpp
1 Below line 362 (1.1) / 369 (1.2) change this:

// Set our working directory.
Torque::FS::SetCwd( "game:/" );

// Set our working directory.
Platform::setCurrentDirectory( Platform::getMainDotCsDir() );

#ifdef TORQUE_PLAYER

to this:

// Set our working directory.
Torque::FS::SetCwd( "game:/" );

// Set our working directory.
if(strncmp(Platform::getExecutableName(),"python",6) == 0){
   // if being launched from Python
   Platform::setMainDotCsDir( Platform::getCurrentDirectory() );
   Platform::setCurrentDirectory( Platform::getCurrentDirectory() );
   Torque::FS::Unmount( "game" );
   Torque::FS::Mount( "game", Platform::FS::createNativeFS( ( const char* ) Platform::getCurrentDirectory() ) );
}else{
   // if launched from Torque Toolbox or the executable wrapper for the DLL
   Platform::setCurrentDirectory( Platform::getMainDotCsDir() );
}

#ifdef TORQUE_PLAYER

Running Under Python Control
Once the changes are in place it should compile without errors. Now you can create a Python script to control the engine. Here are some simple scripts to get you started.

Script Test1:

# script test1

import sys, os
import pytorque
from pytorque import TorqueObject

pytorque.initialize(len(sys.argv),sys.argv)

while pytorque.tick():

    pass

pytorque.shutdown()

Script Test2:

# script test2

import sys, os
import pytorque
from pytorque import TorqueObject

pytorque.initialize(len(sys.argv),sys.argv)

pytorque.evaluate("""
    // this will print to the console
    for(%count = 0; %count < 10; %count++) 
    {
       echo(%count);
    }
    """)

while pytorque.tick():
    
    pass

pytorque.shutdown()

Script Test3 (add this before the while loop of the previous script:

def runonce():
    pytorque.evaluate("""
    new GuiBitmapButtonCtrl(MyButton) {
     profile = "GuiButtonProfile";
     horizSizing = "right";
     vertSizing = "bottom";
     position = "404 361";
     extent = "285 85";
     minExtent = "8 2";
     visible = "1";
     text = "Button";
     groupNum = "-1";
     buttonType = "PushButton";
     bitmap = "./button";
     helpTag = "0";
    };""")
    
    button = TorqueObject("MyButton")

    def OnMyButton(value):
        # this will print to the Python console, not the Torque console
        print "Button pushed with value",value
    pytorque.export(OnMyButton,"MyButton","OnButton","Example button command",1,1)
    # I don't know why it needs the second parameter here. <Solved>
    #button.command = "MyButton::OnButton(42,1);"
    button.command = "MyButton::OnButton(42);"  # fixed bug that required extra do nothing parameter

    button.performClick()

    pytorque.setglobal("$MyVariable",42)
    print pytorque.getglobal("$MyVariable")
    # this prints to the Torque console
    pytorque.evaluate('echo ("*** Here is your variable:" @ $MyVariable);')
    
    return button

button = runonce()

Conclusion

Now that you have a running PyTorque system with T3D it is time to get to the real work. For instance the engine needs the current main.cs to run properly. Without the canvas does not get initialized and we will get an error message. This happens inside the Torque initialization process which occurs when you run pytorque.initialize. I have started to isolate the needed modules for my own projects to get a barebones system. However that may not really be necessary. You may be able to work within the default framework. Time will tell. If I have to gut the system to determine what is what then that will be a great opportunity to learn the ins and out of the CS code.
So now the ball is in your court. Have fun!

Edits:
1. Fixed some bugs with pytorque.cc. One of them caused crashing.
2. No longer need extra bogus parameter for functions. I believe this was a legacy TGE issue. Not sure.
3. Added new link to TDN article location. Will be adding code there and invite others to contribute code samples as well.
4. Added link to diff file with latest changes. I will add diffs incrementally so if there are issues it will be easy to regress if needed.