Holy cow... (cSharp enhanced Torque)
by Vince Gee · 01/23/2012 (8:32 am) · 22 comments
Happy Monday everyone!
Well, what an incredible journey I've been on! Last year sometime I started working on an MMO project for Torque. Well, I wouldn't call it a MMO, since I never imagine getting the millions of subscribers MMO's get, but hey we will call it a MMO anyway!
So, my responsibility in this project is to provide the business and data layer to torque, other people handle the gui, artwork etc.
Server OS: Windows 2008
Database: MS SQL Server 2008
Ok, so last year I built a somewhat complex method for calling C# from torque using the Com Interface via an Out-Of-Process Com Exe. It worked great initially for loading information about players and parameters. Then I added AI's.
Soon I found that the Com Interface couldn't handle the rapid calls from Torque, slowly but surely the Com Interface would completely crap out and crash the server very very bad.
So this weekend, I was pretty much depressed, if I couldn't figure out a way to make this work, months worth of work writing the csharp Dll that the game uses for the Business Layer and Data Layer would be thrown away. It wasn't looking good for my project
In my depression I started reviewing how the pyTorque solution worked, going over the code changes to Torque and trying to get a grasp on the solution as a whole.
Now, for those who don't know me, my Day Job is a business programmer for a Brick and Mortar. I rarely have to do any coding at the nuts and bolts level of anything. Usually, it's a very mundane job and thus the reason I work on this project.
After doing some reading and finding some interesting articles on the internet, I got a prototype working for something I affectionately call 'csTorque', basically a way to start Torque under the CLR runtime using PinInvokes and provide a series of callbacks to the c#.
My goal, is simple, to provide the same level of functionality (or more) that pyTorque offers for Dot Net.
Imagine, if you will, being able to do 90 percent of your game code in a language like vbscript or csharp (the later being my favorite)
Vince
Well, what an incredible journey I've been on! Last year sometime I started working on an MMO project for Torque. Well, I wouldn't call it a MMO, since I never imagine getting the millions of subscribers MMO's get, but hey we will call it a MMO anyway!
So, my responsibility in this project is to provide the business and data layer to torque, other people handle the gui, artwork etc.
Server OS: Windows 2008
Database: MS SQL Server 2008
Ok, so last year I built a somewhat complex method for calling C# from torque using the Com Interface via an Out-Of-Process Com Exe. It worked great initially for loading information about players and parameters. Then I added AI's.
Soon I found that the Com Interface couldn't handle the rapid calls from Torque, slowly but surely the Com Interface would completely crap out and crash the server very very bad.
So this weekend, I was pretty much depressed, if I couldn't figure out a way to make this work, months worth of work writing the csharp Dll that the game uses for the Business Layer and Data Layer would be thrown away. It wasn't looking good for my project
In my depression I started reviewing how the pyTorque solution worked, going over the code changes to Torque and trying to get a grasp on the solution as a whole.
Now, for those who don't know me, my Day Job is a business programmer for a Brick and Mortar. I rarely have to do any coding at the nuts and bolts level of anything. Usually, it's a very mundane job and thus the reason I work on this project.
After doing some reading and finding some interesting articles on the internet, I got a prototype working for something I affectionately call 'csTorque', basically a way to start Torque under the CLR runtime using PinInvokes and provide a series of callbacks to the c#.
My goal, is simple, to provide the same level of functionality (or more) that pyTorque offers for Dot Net.
Imagine, if you will, being able to do 90 percent of your game code in a language like vbscript or csharp (the later being my favorite)
Vince
About the author
www.winterleafentertainment.com
#2
01/23/2012 (10:04 am)
C# is now my favorite scripting language after using Unity. This would be a great addition to torque. 
#3
I also recently updated pyTorque to work with T3D. The SWIG interface is separate and a lot more sophisticated at this point. I have not released anything on the SWIG stuff yet, but once it is mature enough I will be creating another resource.
One thing that you may run into issues with is callbacks in C#. Someone asked about that and said it was difficult due to C# being "managed code". I don't really understand what that means so maybe you can explain that.
Like Richard said, keep us posted!
01/23/2012 (10:34 am)
As you progress I would like to see how you setup your objects and how you interface them to Torque. I am working through doing the same thing using SWIG. So I think if we compared notes it may help us both out.I also recently updated pyTorque to work with T3D. The SWIG interface is separate and a lot more sophisticated at this point. I have not released anything on the SWIG stuff yet, but once it is mature enough I will be creating another resource.
One thing that you may run into issues with is callbacks in C#. Someone asked about that and said it was difficult due to C# being "managed code". I don't really understand what that means so maybe you can explain that.
Like Richard said, keep us posted!
#4
So... once you register the callback, when you use it at a later time, it might be gone... since the garbage collector already took it to the curb. So you need to suppress garbage collection on any variable(s) you use in communicating with torque.
p.s.
(<- I think I was the guy that asked)
01/23/2012 (10:44 am)
The problem with callbacks to C# is that in "managed" languages we have a garbage collector which is greedy. The mf'er will try and steal any object it thinks is no longer used. So... once you register the callback, when you use it at a later time, it might be gone... since the garbage collector already took it to the curb. So you need to suppress garbage collection on any variable(s) you use in communicating with torque.
p.s.
(<- I think I was the guy that asked)
#5
From the CSharp side, I expose a function called:
String ConsoleCall(string)
Which passes the string to the Con::Execute function and then returns the value back to CSharp.
On the TorqueScript side I wrote a callback to the CSharp which calls a function I wrote, that with the help of a little reflections is able to execute the command.
For example in the Torque Script
MudEngine.Instance_Eval("get_PlayerContainer("" @ %client.Character_ID @ "").Relay_ItemBase("" @ %pk_item_base_id @ "")");
It will look in the class you have registered with csTorque and try and find that function in the csharp, execute the function.
Or, you can do things like...
%result = MudEngine.Instance_Eval("get_PlayerContainer("" @ %client.Character_ID @ "").skill_get("" @ %skill_id @ "")");
or even...
%temptransform = MudEngine.Instance_Eval("get_PlayerContainer("" @ %client.Character_ID @ "").player.l_current_zone_transform");
I guess it's not glamorous, but it is pretty damn effective.
From CSharp to Torque
string qq = "" + '"';
string s = "onPropertyChanged(" + qq + torque_object_id + qq + "," + qq + property + qq + "," + qq + value + qq + ");";
m_torquesupport.ConsoleCall(s);
01/23/2012 (10:49 am)
I'm also taking the easy way on hooking this mess together.From the CSharp side, I expose a function called:
String ConsoleCall(string)
Which passes the string to the Con::Execute function and then returns the value back to CSharp.
On the TorqueScript side I wrote a callback to the CSharp which calls a function I wrote, that with the help of a little reflections is able to execute the command.
For example in the Torque Script
MudEngine.Instance_Eval("get_PlayerContainer("" @ %client.Character_ID @ "").Relay_ItemBase("" @ %pk_item_base_id @ "")");
It will look in the class you have registered with csTorque and try and find that function in the csharp, execute the function.
Or, you can do things like...
%result = MudEngine.Instance_Eval("get_PlayerContainer("" @ %client.Character_ID @ "").skill_get("" @ %skill_id @ "")");
or even...
%temptransform = MudEngine.Instance_Eval("get_PlayerContainer("" @ %client.Character_ID @ "").player.l_current_zone_transform");
I guess it's not glamorous, but it is pretty damn effective.
From CSharp to Torque
string qq = "" + '"';
string s = "onPropertyChanged(" + qq + torque_object_id + qq + "," + qq + property + qq + "," + qq + value + qq + ");";
m_torquesupport.ConsoleCall(s);
#6
01/23/2012 (12:04 pm)
very nifty, keep us updated on this, it looks like it has some really good potential. 
#7
Open a Torque 3D DLL in a tool like Dependcy Walker. Note that every export to TorqueScript is also exposed as a external on the DLL.
If you combine that with the information produced by the XML dump of console methods in Engine/source/console/consoleXMLExport.cpp... you should be able to write a tool to generate C# code that calls methods on the Torque DLL. This way C# code can create and call Torque objects in a natural way.
You can also look at CXXI.
01/23/2012 (1:13 pm)
@VinceOpen a Torque 3D DLL in a tool like Dependcy Walker. Note that every export to TorqueScript is also exposed as a external on the DLL.
If you combine that with the information produced by the XML dump of console methods in Engine/source/console/consoleXMLExport.cpp... you should be able to write a tool to generate C# code that calls methods on the Torque DLL. This way C# code can create and call Torque objects in a natural way.
You can also look at CXXI.
#8
As much as I would like to go through and link them all so it's similar to T3d calls, I just don't have the time. I figure, once I get all the bugs out of this thing (or at least where it doesn't crash right away) I will put it up as a resource to show people the basics.
Then,
If an enterprising person wants to flesh it out, more power to them. Sometimes I find that if you give people the start, they are more than happy to finish it.
01/23/2012 (3:49 pm)
Tom,As much as I would like to go through and link them all so it's similar to T3d calls, I just don't have the time. I figure, once I get all the bugs out of this thing (or at least where it doesn't crash right away) I will put it up as a resource to show people the basics.
Then,
If an enterprising person wants to flesh it out, more power to them. Sometimes I find that if you give people the start, they are more than happy to finish it.
#9
Why didn't you post that in my threads earlier?! I will definitely be looking at that consoleXMLExport.cpp! Thanks for mentioning it.
@Vince,
I was wondering if that was you. Yeah, Python also has a garbage collector. That is part of the reason to use SWIG was to help from a conceptual point of view. SWIG has helped me learn a lot about incrementing and decrementing references to Python objects. So it sounds very similar to C# in design.
Someone else chimed in on the console access and found it to be less that speedy. The problem is resolves function calls using string processing. So places where you need speed you will want callbacks if possible. Also, FWIW, SWIG also has a C# interface I believe. It might be worth taking a look at "how they did it" so you can get some ideas for solving some of the harder parts of this effort. That is why I originally looked at SWIG. Overall it has saved me a LOT of time by doing so.
01/23/2012 (4:18 pm)
@Tom!!!Why didn't you post that in my threads earlier?! I will definitely be looking at that consoleXMLExport.cpp! Thanks for mentioning it.
@Vince,
I was wondering if that was you. Yeah, Python also has a garbage collector. That is part of the reason to use SWIG was to help from a conceptual point of view. SWIG has helped me learn a lot about incrementing and decrementing references to Python objects. So it sounds very similar to C# in design.
Someone else chimed in on the console access and found it to be less that speedy. The problem is resolves function calls using string processing. So places where you need speed you will want callbacks if possible. Also, FWIW, SWIG also has a C# interface I believe. It might be worth taking a look at "how they did it" so you can get some ideas for solving some of the harder parts of this effort. That is why I originally looked at SWIG. Overall it has saved me a LOT of time by doing so.
#10
01/23/2012 (6:31 pm)
And Lua - No kidding, Tom, that is a handy piece of info!
#11
My point was is that with that XML dump of classes, function names, and function arguments you can write a small C# program that can automatically build C# wrappers to Torque objects exposed by the DLL.
01/23/2012 (6:57 pm)
@VinceMy point was is that with that XML dump of classes, function names, and function arguments you can write a small C# program that can automatically build C# wrappers to Torque objects exposed by the DLL.
#12
It does not look like it is currently compilable in version 1.2. It will be a good reference for parsing the console objects and namespaces however. Thanks for the information!
01/24/2012 (12:46 am)
@Tom,It does not look like it is currently compilable in version 1.2. It will be a good reference for parsing the console objects and namespaces however. Thanks for the information!
#13
Ok, I see what your saying Tom, I've just been behind the 8 ball of late with our project switching everything over. I have a valid solution now that works very effectively with torque, just did a stability test and the engine passed with flying colors.
I'll take a peak and see what I can find with writing a simple code gen for that class, no promises though :)
@Frank,
CSharp uses a "pass and mark" memory manager, so the only control you have over it is to either tell it to not destroy an object, or not. So you have to think about it a bit more. The other thing that kinda added some bugs in, was that if I tried to make a console call while I was processing a tick the server would crash.
So, I put a lock in place in the stub function so that I only access the Torque object in a linear fashion, no two calls can call torque at the same time. That seemed to really stabilize the server. Was an easy fix but took me a while of pulling my hair out to find it.
01/24/2012 (6:50 am)
@Tom,Ok, I see what your saying Tom, I've just been behind the 8 ball of late with our project switching everything over. I have a valid solution now that works very effectively with torque, just did a stability test and the engine passed with flying colors.
I'll take a peak and see what I can find with writing a simple code gen for that class, no promises though :)
@Frank,
CSharp uses a "pass and mark" memory manager, so the only control you have over it is to either tell it to not destroy an object, or not. So you have to think about it a bit more. The other thing that kinda added some bugs in, was that if I tried to make a console call while I was processing a tick the server would crash.
So, I put a lock in place in the stub function so that I only access the Torque object in a linear fashion, no two calls can call torque at the same time. That seemed to really stabilize the server. Was an easy fix but took me a while of pulling my hair out to find it.
#15
I have not run into that issue yet with the calls. Hmmm, I will keep a look out for it. That must be a threading issue there. I wonder why I have not run into it? I will have to try and produce that situation to see if I can. Then I will know if it is an issue. Of course I have been using Queues that have the threading semantics built in, but still I should have seen that.
Thanks for the heads up!
01/24/2012 (11:39 pm)
@Vince,I have not run into that issue yet with the calls. Hmmm, I will keep a look out for it. That must be a threading issue there. I wonder why I have not run into it? I will have to try and produce that situation to see if I can. Then I will know if it is an issue. Of course I have been using Queues that have the threading semantics built in, but still I should have seen that.
Thanks for the heads up!
#16
Does Swig set up multi threaded stubs? I've not worked with swig enough to know how it sets every thing up.
01/25/2012 (3:28 am)
@Frank,Does Swig set up multi threaded stubs? I've not worked with swig enough to know how it sets every thing up.
#17
I only ever access the console/torque object with one thread. It does not matter what the tick is doing. I don't know think SWIG does anything with threads that I know of. I did run into issues if multiple threads accessed the Torque object. So maybe I have seen what you are talking about. One thing about the tick is that I release the Python GIL while the tick is processing so my python code can run at full speed. I do all sorts of accessing the Torque object during that time, but only through 1 thread. So that is probably the reason I am not seeing what you had issues with.
01/25/2012 (1:38 pm)
@Vince,I only ever access the console/torque object with one thread. It does not matter what the tick is doing. I don't know think SWIG does anything with threads that I know of. I did run into issues if multiple threads accessed the Torque object. So maybe I have seen what you are talking about. One thing about the tick is that I release the Python GIL while the tick is processing so my python code can run at full speed. I do all sorts of accessing the Torque object during that time, but only through 1 thread. So that is probably the reason I am not seeing what you had issues with.
#18
01/26/2012 (7:36 pm)
Tom, I'm glad you posted that. I came in to say that both the pyTorque solution and what was being suggested here are both inelegant solutions, and that it would be much more sensible to provide SWIG interfaces to the T3D Console system.
#19
Huh? What exactly does that mean?
SWIG does not talk to DLLs like that. It is compiled into the code itself and creates a DLL interface. Unless I completely missed that part in the docs. SWIG creates an interface to the classes, variables, objects, functions defined in the source code.
Or are you thinking that it would go into the external exe that Torque defines? Please explain as I am not "getting it". Thanks in advance.
@Tom (the first Tom, haha),
I just got a chance to look at the dependency walker program. It is way cool and useful. Thanks!
01/26/2012 (8:32 pm)
@Thomas,Huh? What exactly does that mean?
SWIG does not talk to DLLs like that. It is compiled into the code itself and creates a DLL interface. Unless I completely missed that part in the docs. SWIG creates an interface to the classes, variables, objects, functions defined in the source code.
Or are you thinking that it would go into the external exe that Torque defines? Please explain as I am not "getting it". Thanks in advance.
@Tom (the first Tom, haha),
I just got a chance to look at the dependency walker program. It is way cool and useful. Thanks!
#20
DW is an interesting program. Although because of the way the Torque DLL is compiled it is only showing the function names and not the parameters. Most of the ones that deal with the console are functions I had already found. Many of the ones I need and am using are not exposed. Or at least I don't recognize where they are exposed.
Some of the functions I had issues with using and had to rewrite my own version to compensate. One such limitation is calling Python callback functions. If I create a callback function that I register in the console that is associated with a namespace I cannot get the correct namespace at the time of the call. So I had a choice, either have fixed Python callbacks with know namespace they would be called in or create a new console function type that will get called and inject the namespace into the parameters provided to the callback code. This could not be done using just the external interface. BTW, this was originally solved in pyTorque using a global function that grabbed the namespace out of the console code before it called the callback. I did the same thing without a global and a new console function type.
I don't really know I am building my SWIG interface the "right" way, but I am learning a lot about the engine by doing so.
Anyway, sorry for hijacking this thread. I hope this discussion is fruitful for everyone.
01/26/2012 (11:37 pm)
@Tom,DW is an interesting program. Although because of the way the Torque DLL is compiled it is only showing the function names and not the parameters. Most of the ones that deal with the console are functions I had already found. Many of the ones I need and am using are not exposed. Or at least I don't recognize where they are exposed.
Some of the functions I had issues with using and had to rewrite my own version to compensate. One such limitation is calling Python callback functions. If I create a callback function that I register in the console that is associated with a namespace I cannot get the correct namespace at the time of the call. So I had a choice, either have fixed Python callbacks with know namespace they would be called in or create a new console function type that will get called and inject the namespace into the parameters provided to the callback code. This could not be done using just the external interface. BTW, this was originally solved in pyTorque using a global function that grabbed the namespace out of the console code before it called the callback. I did the same thing without a global and a new console function type.
I don't really know I am building my SWIG interface the "right" way, but I am learning a lot about the engine by doing so.
Anyway, sorry for hijacking this thread. I hope this discussion is fruitful for everyone.
Torque Owner Richard Ranft
Roostertail Games