Torque, Python and the GIL
by Demolishun · 11/28/2011 (1:43 am) · 10 comments
I love to program Python. It is truly a joy to play with and make a living with.
I love to program Torque. It is truly a joy to play with and make a living with.
Notice how the two sentences are very similar and work well in parallel. Not necessarily so when it comes to running the two together. You see at times they access the same resources and must be controlled. Well, in Python the control method is through the GIL.
I will save you time if you are just reading this for the entertainment value. The GIL makes sure only one thread is using the Python resources at any one time. So each thread will run happily along and run just fine until it needs a Python resource. Then it must acquire the GIL so it can use those resources. Well, what the heck does this have to do with Torque?! Nothing, normally...
Welcome to the off-normal PyTorque. This basically extends Python with the Torque engine. That is just to fill in for those who don't know about it. This is where it gets really interesting.
So we have Torque which now appears as a library extension to Python. Python calls the Torque engine through an API. What happens when we put Torque in a Python thread so we can run Python code is parallel to the Torque code? Well it runs really slow on the Python side. What!? How can this be? They are in separate threads? Ummm, yes, and ummm, there is an issue. The GIL. The what? You see the PyTorque resource was not written to account for running the Torque code in parallel. It did not isolate the portions of code that may block with calls to release the GIL.
That was my ahah! moment. I had Torque running is a separate thread and the Python code in the main thread was running painfully slow. Then I asked the question: How can the Torque code be blocking the Python code? They are in separate threads. So a bunch of reading and literally less than 15 lines of code later I was able to get Torque and Python running in parallel each at full speed. It is a beautiful thing to watch your Python loop screaming by at top speed and shooting the crap out of bad dudes in Torque. I have never enjoyed 'print' functions so much.
I will be updating the resource at some point to include the threading changes and creating more examples of running things in parallel with the Torque engine. This explains why my original wx implementation did not play well with Torque. Ahah! Yes, another ahah! moment. ;)
I love to program Torque. It is truly a joy to play with and make a living with.
Notice how the two sentences are very similar and work well in parallel. Not necessarily so when it comes to running the two together. You see at times they access the same resources and must be controlled. Well, in Python the control method is through the GIL.
I will save you time if you are just reading this for the entertainment value. The GIL makes sure only one thread is using the Python resources at any one time. So each thread will run happily along and run just fine until it needs a Python resource. Then it must acquire the GIL so it can use those resources. Well, what the heck does this have to do with Torque?! Nothing, normally...
Welcome to the off-normal PyTorque. This basically extends Python with the Torque engine. That is just to fill in for those who don't know about it. This is where it gets really interesting.
So we have Torque which now appears as a library extension to Python. Python calls the Torque engine through an API. What happens when we put Torque in a Python thread so we can run Python code is parallel to the Torque code? Well it runs really slow on the Python side. What!? How can this be? They are in separate threads? Ummm, yes, and ummm, there is an issue. The GIL. The what? You see the PyTorque resource was not written to account for running the Torque code in parallel. It did not isolate the portions of code that may block with calls to release the GIL.
That was my ahah! moment. I had Torque running is a separate thread and the Python code in the main thread was running painfully slow. Then I asked the question: How can the Torque code be blocking the Python code? They are in separate threads. So a bunch of reading and literally less than 15 lines of code later I was able to get Torque and Python running in parallel each at full speed. It is a beautiful thing to watch your Python loop screaming by at top speed and shooting the crap out of bad dudes in Torque. I have never enjoyed 'print' functions so much.
I will be updating the resource at some point to include the threading changes and creating more examples of running things in parallel with the Torque engine. This explains why my original wx implementation did not play well with Torque. Ahah! Yes, another ahah! moment. ;)
About the author
I love programming, I love programming things that go click, whirr, boom. For organized T3D Links visit: http://demolishun.com/?page_id=67
#2
That is the question I have the most difficulty with. So it would probably be best answered like this:
Standard Library
and
Package Index
For the package index notice the number at the top of the page in bold.
So the best I can answer is "a lot". My personal interests are lxml, web services (like oauth), web servers, databases, dynamic game environments using data from other sources (auto GIS retrieval) and autogenerated.
Other things that interest me, but I have not added the ability to do so yet: transfer of binary data between Python and Torque. Like 3D objects, textures, sounds, etc. One example is a synth library in Python. I could transfer it by file, but going through memory accessible by both Python and Torque would be better. Also, imagine bringing in something like internet radio or midi streams. I am sure with some work all of this would be possible with TS, but Python code and packages are battle tested and already built.
11/28/2011 (10:53 am)
@Daniel,That is the question I have the most difficulty with. So it would probably be best answered like this:
Standard Library
and
Package Index
For the package index notice the number at the top of the page in bold.
So the best I can answer is "a lot". My personal interests are lxml, web services (like oauth), web servers, databases, dynamic game environments using data from other sources (auto GIS retrieval) and autogenerated.
Other things that interest me, but I have not added the ability to do so yet: transfer of binary data between Python and Torque. Like 3D objects, textures, sounds, etc. One example is a synth library in Python. I could transfer it by file, but going through memory accessible by both Python and Torque would be better. Also, imagine bringing in something like internet radio or midi streams. I am sure with some work all of this would be possible with TS, but Python code and packages are battle tested and already built.
#3
Recently, I've been thinking about restarting the game, but this time with C# and the appropriate libraries as the back end. But Python always feels like a good, trusted friend. I look forward to seeing what changes you've put in place to better work with the GIL.
- Dave
11/28/2011 (12:20 pm)
That's really interesting. Years ago, I had Torque tied into Stackless Python for all of my back end server needs for a space game I was working on. I even turned the commercial version of TNL into a Python library to make use of its UDP networking magic. My greatest fear was always Python's lack of hardware thread support and being able to take advantage of multi-core processors. I ended up putting the project on hold once I had the back end good-to-go and realized the enormity of the content requirements for the game.Recently, I've been thinking about restarting the game, but this time with C# and the appropriate libraries as the back end. But Python always feels like a good, trusted friend. I look forward to seeing what changes you've put in place to better work with the GIL.
- Dave
#4
The changes were really minimal. It was more understanding the function of the GIL than anything. Python does support threads and actually creates threads using the OS specific thread control.
I will be updating the resource to show the needed changes. They are not big. They just isolate the code to release the GIL and then reacquire the GIL when it comes back into Python space.
11/28/2011 (12:50 pm)
@Dave,The changes were really minimal. It was more understanding the function of the GIL than anything. Python does support threads and actually creates threads using the OS specific thread control.
I will be updating the resource to show the needed changes. They are not big. They just isolate the code to release the GIL and then reacquire the GIL when it comes back into Python space.
#6
Check out this link:
docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock
Edit:
@David,
I see what you are asking:
Notice it says executing 'Python' code. If you are using an app that attaches code and extends python then it is not 'python' code except where it calls the Python API to communicate with the interpreter. So if you properly isolate the code that will run outside of the python api then you will get your parallel execution. A large number of the Python libraries are not pure python and thus would be able to operate is parallel very nicely. It really is no different than other synchronization methods. In other threaded programs you still have times when it has to wait for a resource to become available.
I dunno, I have used Python threads to great success with pure Python code. It seems to work 'good enough'. Once could always spawn a daemon app and communicate with that if absolute parallel execution is required.
11/28/2011 (5:03 pm)
This is for 2.x. Maybe I don't understand what you mean by hardware level.Check out this link:
docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock
Edit:
@David,
I see what you are asking:
Quote:CPython implementation detail: Due to the Global Interpreter Lock, in CPython only one thread can execute Python code at once
Notice it says executing 'Python' code. If you are using an app that attaches code and extends python then it is not 'python' code except where it calls the Python API to communicate with the interpreter. So if you properly isolate the code that will run outside of the python api then you will get your parallel execution. A large number of the Python libraries are not pure python and thus would be able to operate is parallel very nicely. It really is no different than other synchronization methods. In other threaded programs you still have times when it has to wait for a resource to become available.
I dunno, I have used Python threads to great success with pure Python code. It seems to work 'good enough'. Once could always spawn a daemon app and communicate with that if absolute parallel execution is required.
#7
Yes, your added comment is what I was referring to. I was looking down the road and wanting to have my Python code take advantage of multicore processors. I had all of my server and game logic under Python, and certainly using Python threads, or in my case tasklets under Stackless, helps feel like you have concurrent tasks running, but in the end you're still trapped to that one physical hardware thread.
And as you suggest, I had my networking layer release and reacquire the GIL so it could run concurrently with my Python code. And certainly I could have broken out parts of my Python code into separate native code libraries to get that extra processing power of additional cores. But at some point it seemed to be fighting against rather than working with the system. I know the CCP folks have had to do the same thing for EVE Online: break out code that can run concurrently into separate native code libraries to make use of the extra processing power on their servers.
Sorry, I didn't mean to derail you blog or harp on Python. I still use it as a utility language whenever I can and it makes a great scripting replacement for Torque, especially, as you've pointed out, given the number of great libraries for it. Doing something like adding a web server to a Torque game server to allow for control using a browser, or retrieving statistics using REST are extremely easy and very useful.
- Dave
11/30/2011 (8:20 am)
Hey Frank.Yes, your added comment is what I was referring to. I was looking down the road and wanting to have my Python code take advantage of multicore processors. I had all of my server and game logic under Python, and certainly using Python threads, or in my case tasklets under Stackless, helps feel like you have concurrent tasks running, but in the end you're still trapped to that one physical hardware thread.
And as you suggest, I had my networking layer release and reacquire the GIL so it could run concurrently with my Python code. And certainly I could have broken out parts of my Python code into separate native code libraries to get that extra processing power of additional cores. But at some point it seemed to be fighting against rather than working with the system. I know the CCP folks have had to do the same thing for EVE Online: break out code that can run concurrently into separate native code libraries to make use of the extra processing power on their servers.
Sorry, I didn't mean to derail you blog or harp on Python. I still use it as a utility language whenever I can and it makes a great scripting replacement for Torque, especially, as you've pointed out, given the number of great libraries for it. Doing something like adding a web server to a Torque game server to allow for control using a browser, or retrieving statistics using REST are extremely easy and very useful.
- Dave
#8
No derailment here. This is a fascinating topic. Python is a tool and like all tools have limitations. I don't see the current threading implementation as a limitation as you can always create a *nix style daemon to separate out the processes. At that point you lose shared memory access, but nothing is for free.
I did a quick search on Google and found this: www.parallelpython.com/
It looks like it creates parallel processes with network transparency. Obviously not sharing memory, but a SQL database would be ideal. More like a supercomputer.
It also talks a little bit about the GIL. It says the GIL cannot execute Python bytecode in parallel. Which is kind of what I gathered from the other sites. I may have to look at it because you could potentially keep one thread attached to Torque and have that thread manage the rest of the servers and their processing power. That might make for a very nice server implementation for a Torque server. Also, another application would be running the Torque server and keeping in sync with it in case of a need for a hot fail over. Neat stuff!
11/30/2011 (7:48 pm)
@Dave,No derailment here. This is a fascinating topic. Python is a tool and like all tools have limitations. I don't see the current threading implementation as a limitation as you can always create a *nix style daemon to separate out the processes. At that point you lose shared memory access, but nothing is for free.
I did a quick search on Google and found this: www.parallelpython.com/
It looks like it creates parallel processes with network transparency. Obviously not sharing memory, but a SQL database would be ideal. More like a supercomputer.
It also talks a little bit about the GIL. It says the GIL cannot execute Python bytecode in parallel. Which is kind of what I gathered from the other sites. I may have to look at it because you could potentially keep one thread attached to Torque and have that thread manage the rest of the servers and their processing power. That might make for a very nice server implementation for a Torque server. Also, another application would be running the Torque server and keeping in sync with it in case of a need for a hot fail over. Neat stuff!
#9
11/30/2011 (8:23 pm)
Was the Edit #4 on the Resource the new changes?
#10
Edit: Also, I am going to explore using SWIG for this redo the entire extension. That way it can be generated and will be more cross platform compatible. At that point it will not longer depend upon the original PG code.
12/01/2011 (6:21 pm)
The diff has all the changes from the original in it. So if you take the file from PG and apply the diff you will have everything I have so far. You could just pick out the changes for the GIL optimizations if you already have implemented part of the resource.Edit: Also, I am going to explore using SWIG for this redo the entire extension. That way it can be generated and will be more cross platform compatible. At that point it will not longer depend upon the original PG code.
Torque Owner Daniel Buckmaster
T3D Steering Committee