Game Development Community

PyTGE (also PyTSE) - Python Bindings

by Prairie Games · in Torque Game Engine · 07/12/2006 (2:38 am) · 101 replies

I've had a number of requests for this so here it is sooner rather than later... Python bindings for TGE/TSE and probably TGB:

http://www.prairiegames.com/pytse10a.zip

It's called PyTSE, though there isn't any TSE specific source in it. So, it should build with no modifications with TGE 1.4

This hasn't been field tested, though it should be more or less bug free. Here's the text from the readme.txt, it's sparse I know. Time is a very limited commodity :|

Quote:This is a new TGE/TSE (and probably TGB) Python binding I have been working on. You can do with it what you like.

I don't have any time to document it. Though, the source file is only 16k, so it should be pretty clear.

I've also included a diff of source changes. They should be pretty close to the current HEAD revision.

You need to change your build target to a shared library (a dll on windows, also change to multithreaded dll code generation for this platform)

Here are some minimal docs in the forum of example usage, replace TSE with TGE if that is your desire:

[b]#--- TSE Python Module Example ---[/b]

[b]#TSE as a standard Python extension (no longer a executable)[/b]
import pytse

[b]#initialize pytse, this also executes main.cs and the .cs packages[/b]
pytse.initialize()

[b]#example of executing a script file[/b]
f = file("myscript.cs","rb")
script = f.read()
f.close()
pytse.evaluate(script)

[b]#or, just generate the cs code right inside Python![/b] 
pytse.evaluate("""
new GuiBitmapButtonCtrl(MyButton) {
 profile = "GuiButtonProfile";
 horizSizing = "right";
 vertSizing = "bottom";
 position = "404 361";
 extent = "285 85";
 minExtent = "8 2";
 visible = "1";
 text = "Button";
 groupNum = "-1";
 buttonType = "PushButton";
 bitmap = "./button";
 helpTag = "0";
};""")

[b]#it's easy to grab a reference to the button we created[/b]
button = TSEObject("MyButton")

[b]#buttons are kind of worthless without commands.  Let's make one:[/b]
def OnMyButton(value):
    print "Button pushed with value",value
    
[b]#export the function to the console system in much the same way the C++ system does...
#we also support optional namespaces, usage documentation, and min/max args[/b]
pytse.export(OnMyButton,"MyButton","OnButton","Example button command",1,1)

[b]#we can get and set fields (including dynamic fields).  We'll set our button's command:[/b]
button.command = "MyButton::OnButton(42);"

[b]#we can call console methods on our TSEObjects... So, let's simulate a button click.  
#the OnMyButton function will be called with the value 42 :)[/b]
button.performClick()

[i]#note that getting an object reference to the button and setting the command like this is 
purely for illustration. You can also: command = "MyButton::OnButton(42);" in the evaluated code.[/i]

[b]#moving on, we can get and set global variables[/b]
pytse.setglobal("$MyVariable",42)
print pytse.getglobal("$MyVariable")
pytse.evaluate('echo ("*** Here is your variable:" @ $MyVariable);')

[b]#the main loop is broken out and can be combined with other frameworks rather easily[/b]
while pytse.tick():
    pass

[b]#cleanup pytse.. goodbye![/b]
pytse.shutdown()


-Josh Ritter
Prairie Games, Inc
#41
07/21/2006 (9:14 am)
I would love to only need 200 lines of TorqueScript. :) I'm sure that you are right when you say there's no real need to write dynamic fields. I'm going through the tutorials to learn TGE, so these are things I ran into while doing that and certainly isn't representative of how I would do things if I knew my way around TGE better.

Thanks again for sharing this fantastic code!
#42
07/21/2006 (8:06 pm)
Well, my status under OSX so far:

I ended up moving the OpenAL, Ogg, Vorbis, and Theora frameworks to the system level, which fixed the problem of trying to reference said frameworks. The library builds cleanly, however I ended up commenting out all the stuff that handles command line arguments in TSE_Initialize() at the moment because they are all Windows specific calls.

Edit:

Ok I see, TSE_Initialize() basically copies the WinMain() code from platformWin32.cc, which definately doesn't seem to be as simple when looking at platformMacCarb.cc :(

Anyways, even with the commented out command line bits and not handling the app window, the library does seem to kind of work.

boju:~/Documents/Code/gamedev/pytest bojo$ python game.py 
game.py:4: RuntimeWarning: Python C API version mismatch for module pytse: This Python has API version 1013, module pytse has version 1012.
  import pytse
Console Initialized.
% Error: Unable to load any specified mods
--------- Parsing Arguments ---------
Engine initialized... 
Warning: (/Users/bojo/Documents/Code/gamedev/Torque-1.4-pytse/pb/../engine/gui/core/guiTypes.cc @ 347) GuiControlProfile: requested gui profile (GuiButtonProfile) does not exist.
Fatal: (/Users/bojo/Documents/Code/gamedev/Torque-1.4-pytse/pb/../engine/gui/core/guiTypes.cc @ 351) GuiControlProfile: unable to find specified profile (GuiButtonProfile) and GuiDefaultProfile does not exist!
DEBUG_BREAK!
DEBUG_BREAK!  
Button pushed with value 42
42
*** Here is your variable:42
#43
07/24/2006 (11:37 pm)
Ok, I think I have things working at some level under OSX, although I could be wrong. There's still a few things to fix I'm sure, but here we go...

First I had to get the headers right:

// Add these
+ // NOTE: Placing system headers before Torque's platform.h will work around the Torque-Redefines-New problems.
+ #include <Carbon/Carbon.h>
+ #include <CoreServices/CoreServices.h>

#include "platform/platform.h"
...
// Change these so we are using the right platform.  
// The above includes fix the 'new' problem, which I wasn't aware of the first time through
//#include "platformWin32/platformWin32.h"
+ #include "platformMacCarb/platformMacCarb.h"

My TSE_Initialize() looks like this (minus any arg handling as you can see, oh well):

static int TSE_Initialize()
{

   LinkConsoleFunctions = true;
   
   createFontInit();
   
   S32 ret = Game->initialize(NULL, NULL);
   
   gPyTSEInitialized = true;

   return 1;

}

And I had a problem with getting the correct path to load. Turns out it was trying to load main.cs from the Python execution point, rather than the loaded library's directory. So in macCarbFileio.cc I added these rather hacky lines:

StringTableEntry Platform::getWorkingDirectory()

{
   + // PyTSE hack
   + cwd = getcwd(NULL, 0);
   + chdir(cwd);

   if(!cwd)
   {

Things seem to work from there. The example code Josh pasted above works (at least as far as terminal output is concerned), so I'll have to dive in further later this week and actually get things rolling. As you can see I am studying for all my finals that are this week... :)
#44
07/25/2006 (1:16 pm)
Brian, I haven't tried what you suggested yet, but I definitely will. Good luck with further improvements, and thanks so much for blazing the trail! :)
#45
07/25/2006 (3:58 pm)
Thanks Jason, although like I said I'm not sure if it's the right approach or not. Overriding that cwd variable kind of bugs me, and there's still no command line handling. I learned quite a bit about some of the platform inner workings though, and that's important.

However, now I am trying to figure out how to actually get things to init purely from python. I was tinkering a bit with it last night, but I can't seem to immitate my .cs init scripts at a rudimentary level.
#46
07/26/2006 (7:28 pm)
I've been working with this a bit now, after first wrapping my brain around TGE enough to do something besides read tutorials and forum discussions. :) But I was curious what is the best way to deal with the nightmare that is TorqueScript's use of strings all over the place?

For instance, to do something with a position (ie, create an offset based on current position of the player), you have to get the space-delimited string containing the 3 values, then split them, then cast them to floats, then do your math, then cast them to strings and concatenate them back into a string of 3 values to pass to another Torque function. This is horrific to me.

Is there a better way? Josh, how did you handle that in MoM, if I may ask?
#47
07/26/2006 (9:09 pm)
Yes, there is a better way. Though, what you descibe is exactly what you do C++ side when working with TorqueScript. In fact, working with this stuff in TorqueScript itself is quite gross:

%pos    = getWords(%obj.getTransform(), 0, 2);
%oldPos = %pos;
%vec[0] = " 0  0  1";
%vec[1] = " 0  0  1";
%vec[2] = " 0  0 -1";
%vec[3] = " 1  0  0";
%vec[4] = "-1  0  0";
%impulseVec  = "0 0 0";
%vec[0] = MatrixMulVector( %obj.getTransform(), %vec[0]);

Anyway, what I did with MoM are interceptions like the following:

if (!dStricmp(argv[0],"getPosition"))
{
   F32 pos[3];
   dSscanf(ret,"%f %f %f",&pos[0],&pos[1],&pos[2]);

   PyObject* tuple = PyTuple_New(3);
   PyTuple_SetItem(tuple,0,PyFloat_FromDouble(pos[0]));
   PyTuple_SetItem(tuple,1,PyFloat_FromDouble(pos[1]));
   PyTuple_SetItem(tuple,2,PyFloat_FromDouble(pos[2]));

   return tuple;
}

and then in Python we can just:

pos = obj.getPosition()
print pos[0],pos[1],pos[2]

or

x,y,z = obj.getPosition()
print x,y,z

If you aren't overly worried about performance due to a call being infrequent or whatever, you could just write a convenience function or method in Python.

-JR
#48
07/26/2006 (11:07 pm)
Thanks Josh, you got me to buy TGB. :)

Tomorrow I start the work of getting this to go. Thanks again!
#49
07/27/2006 (6:59 am)
Oh that's much nicer, Josh! Thanks for the info.

Edit: Just wanted to say that it works great and is much nicer to use.
#50
07/27/2006 (10:49 am)
Since I'm never satisfied, I wanted to be able to use that convenient conversion on anything that returns 3 floats. But I didn't want a ton of if's in the code which would slow it down. So using the "switchScanner" tool, I added all the function names I wanted to intercept and now they all return tuples of 3 floats. This is not a general solution, but it will make life easier for those cases. This would also be possible to utilize for TGB and TSE, of course.

I added
token t;
while ( (t = TGEscanner((char **)&argv[0])) > TOKEN_NOERROR )
{
   F32 pos[3];
   dSscanf(ret,"%f %f %f",&pos[0],&pos[1],&pos[2]);

   PyObject* tuple = PyTuple_New(3);
   PyTuple_SetItem(tuple,0,PyFloat_FromDouble(pos[0]));
   PyTuple_SetItem(tuple,1,PyFloat_FromDouble(pos[1]));
   PyTuple_SetItem(tuple,2,PyFloat_FromDouble(pos[2]));

   return tuple;
}
to PyTSEObject_call() and it seems to work great.

Using the switchScanner, I generated the .c and .h files with:
import switchScanner

s = switchScanner.switchScanner("TGEscanner")

# List all function calls whoose return values we want converted to a Python
# triplet tuple (f, f, f)

# SceneObject / ScriptBase / Player
s.addKeyword( "getPosition" )
s.addKeyword( "getForwardVector" )
s.addKeyword( "getWorldBoxCenter" )
s.addKeyword( "getAIRepairPoint" )
s.addKeyword( "getEyePoint" )
s.addKeyword( "getEyeVector" )
s.addKeyword( "getMuzzlePoint" )
s.addKeyword( "getMuzzleVector" )
s.addKeyword( "getVelocity" )

# AIPlayer
s.addKeyword( "getAimLocation" )
s.addKeyword( "getMoveDestination" )

# Sky
s.addKeyword( "getWindVelocity" )

switchScanner.write()
Then you just add the resulting scanner.h and scanner.c files to the project and recompile. Anything in the list will get intercepted and the return value will become a tuple of 3 floats.

Must say Big Thanks to Larry Hastings, the author of the switchScanner tool.
#51
07/27/2006 (11:10 am)
Hey, that quite awesome! :)
#52
07/27/2006 (11:40 am)
I hope you can use it. I feel better being able to contribute something back.
#53
07/27/2006 (12:03 pm)
I have a problem..

Stupid one I am sure. I've got TGB to compile as a DLL, no problem. I think that part of it is fine.

I then rename it to pytse.pyd as directed. If I copy and paste the example in the OP into a .py file and fix the one syntax error it validates fine.

Here's the problem though - if I drop example.py into the main "games" directory, it will run and give me a nice black TGB screen with nothing happenning. If I kill TGB, both IDLE and the Python shell remain hung and I have to kill them as well.

NB: I should say that I removed the example on how to parse a cs file and even the button code eval on further tests.

If I go to the command line and type:

import pytse
pytse.initialize()

I get the same effect.

What am I doing wrong here? I'm just trying to get a "hello world" going to understand how this works. I don't have to hit some render method or something in the main loop do I?

Moving example.py and the .pyd to one of the example subdirectories I get a quite diffferent result.. I can import, initialize - and nothing happens. I thought initialize loaded and executed main.cs, etc? - shouldn't initialize basically run the game?

EDIT: Just saw a popup that didn't appear before.. Error is:


Fatal - c:\program files\torquegamebuilderpro\engine\source\console\simbase.cc @973
SimObject::object missing call to SimObject::onRemove

I get this after I close TGB and try to get a response for the Python shell.
#54
07/27/2006 (3:07 pm)
Apparently the problem was that TGB needed one more modification, to the DemoGame::tick() method.

Sorted now, thanks Jeremy.
#55
07/27/2006 (3:36 pm)
No problem
#56
07/27/2006 (4:12 pm)
Care to share the love with the rest of us? :)
#57
07/28/2006 (12:54 am)
Don't have a diff, so here's a handmade one:

in engine/source/game/main.cc

DemoGame::tick()

PROFILE_START(TimeManagerProcessMain);
TimeManager::process(); // guaranteed to produce an event
PROFILE_END();
+ PROFILE_START(GameProcessEvents);
+ Game->processEvents();
+ PROFILE_END();
PROFILE_END();
#58
07/28/2006 (1:01 am)
Now that's done, I'll pose a question I posed to Jeremy.. Perhaps someone can enlighten me.

When I put example.py in the TGB games directory (with pytse.pyd and such), it runs TGB just fine. I can run all of my previous and example TGB projects no problem.

However, when I try to use that example.py to run a specific project by putting example.py and pytse.pyd into a specific project directory, I get nothing but a stalled Python. I also copied all of the other dlls in that dir to the project dir and it makes no difference.

I have tried modifying the paths in main.cs and game.cs to be absolute - no effect.

To determine what was going on, I grabbed FilemonNT from Sysinternals and watched what Python did when I ran example.py. It was a lot of stuff. But what it came down to at the end was it finally loaded main.cs - and then stopped. It never even attempted to load the gui or any other .cs files. (Filemon shows all file handle attempts, so it would show a failed attempt to load a file with a bad path, say)

So, what's happenning here? How do I set up a project workspace so it will actually work?

For reference, my example.py is uber-simple.

import pytse
pytse.initialize()

while pytse.tick ()
	pass

pytse.shutdown()
#59
07/28/2006 (9:06 am)
Quote:However, when I try to use that example.py to run a specific project by putting example.py and pytse.pyd into a specific project directory, I get nothing but a stalled Python.
I got TGB running with PyTSE (thanks for posting the last piece of the puzzle for that), though I didn't try to create a new project yet. But you shouldn't need to move those files. Just work from the top level directory, referring to the project as a module. Ie, add empty __init__.py files to the sub-directories so you can import files from there.

Maybe I didn't understand what you want to do, though.
#60
07/28/2006 (10:24 am)
Ya that doesn't work for me I don't think.

Since the main loop is in Python (in this case, in example.py) - I need to be able to execute example.py and have it run the game. Right now I can't.

Someone's gotta know how to setup a project directory for this?

EDIT: Some clarification on the problem:

It seems that when I use python to exec a TS file, it executes it but any further exec() statements in TorqueScript don't work. According to Filemon it's not even trying to grab them with bad paths. I found out that I needed to execute the stuff in Common to get all the TorqueScript features without manually recreating all of that in Python - but unfortunately I can't since the stacked execs() don't work.

I'm going to have to slip some debug stuff in to TGB itself to see what's going wrong.

Surely somebody must've sorted this before and has some ideas?