Using the Torque Platform in a Persistant FPS
by Rob Sandbach · in General Discussion · 09/30/2009 (6:12 pm) · 4 replies
Hello all,
About Me
I have a feeling this may be a long post! Over at indiespring we're planning our "big project". I come from 3-5 years of using torque technology including releasing our recent game Fluttr, helping on the Combat Starter Kit, working on an FPS conversion from a mod and generally prototyping and playing around in Torque as an avid hobbiest. At the scripting level I'm extremely confident and comfortable with TGB, TGE and what I've played with in TGEA. At the C++ level I am a good understanding of torque's methods, have implemented many resources, and make small changes myself. My day job is in digital marketing, so I also have a fairly good understanding of the web & basic server architecture. I have a novice understanding of PHP, apache and the linux commandline and an extremely basic SQL comprehension (INSERT, UPDATE and DELETE). I also have experience with basic visualization through VMWare. IN the development of this project I would have access to experience database architects, and C++ coders to help me with deployment. It's not just me! So that's where I am.
About the Idea
The game that I am planning stems from my personal interests, my vision/understanding of the market place and my experience and access to resources in my day job. I have identified (what I believe to be) a potential niche in a persistent world server which hosts an FPS type game. That is, there are many players playing traditional multiplayer FPS's on non-persistant round based servers of 2-64 players which can be self hosted. There is also a category of FPS's (namely Planetside and WWIIonline) which aim to allow 1000s of concurrent players to war in a persistent, or campaign based, world. I am looking to build somewhat of a blend of these categories. A game in which a players character is persistent, has "statistics" which are remembered between sessions, will appear where they log off and so forth. There would be "one" server which players connect to and play on.
About the Technology
I am planning on building the server on cloud based infrastructure. For the following passage I assume that this is Amazon EC2 + S3 (though this is not 100%). Each Torque instance or zone server runs an "island". As an island fills up and gets close to it's maximum player count torque liases with AWS and spawns a new instance of an island server, which fires up. Some players are then offered an in-game incentive to withdraw from the current island, and drop into the new island to try and fight the battle there. This is the "basic" idea from which I want to build a scalable infrastructure in which costs of running are directly correlated to the number of players. As more players join, we add more islands and the resources available to them. As players leave, we come back down to one. It would not however scale to 1000s of players in one seamless world - I'd like to be realistic in my approach as a small studio. The goal of this post is to understand what is a realistic expectation.
My proposed server infrastructure is below:
There will also be a separate server responsible for patching and update delivery. (Darn my rushing!).
All database queries would go through the "custom" level written in java/python or something similar. These would communicate at the socket level with torque clients, as well as with the database servers. It's apparant that the "database" in that diagram may require replication/load balancing of it's own, but I have folk from my web development team who can handle that, for now I'm happy to assume that I have the actual database scaling covered. I'm more worried about understanding the game side of things.
The Questions
This thread is really to ask questions and gather advice on the technological and infrastructural implications of the game. Although I'd love to hear your feedback about the business model and player's opinions of such a game, if you could email them to me that would be fantastic. Here I'm really interested in answering the following, quite indepth, queries:
1.My first objective/priority is to get a T3D dedicated server instance on a given server running reliably with as many client as it reasonably can. I have read many threads on this and will be implementing the following changes. Tweaks to the networking parameters. Rewrite of the player class, possibly even the shapebase class to make it as lightweight as possible. Removing the sending of datablocks at mission load. Changing most rapid fire weapons to use raycats, not projectiles. Disabling polysoup collision and relying on the old conves collision hulls. No "dynamic physics" like destruction etc. To my questions. a) Are there any other obvious changes I've over looked that could be implemented and b) Aare DIFs my best bet for buildings/infrastructure. Will I get a significant performance gain from disaling poly soup and using them rather than traditional poly-based buildings?
2. In my scenario, is CPU likely to be the limiting factor, or memory? See [url
here for details on what specs an extra large sever instance will be. Is the "CPU Intensive" package going to significantly boost my player counts?
3. I know it's difficult/hard/impossible? to do so, but any rough indication or estimation of approximate player counts following these changes would be really useful to me.
4. When I prototype a T3D server on the instance, what's the best method to test the player count? Short of finding 100s of testers!?
5. Once I can get as many players as I can onto each island, I want to investigate, in the long run, making the transition between small instance, to a large instance on the same island server. At present this requires a reboot on Amazon's behalf (hence at least initially, each server will be running as an extra large instance server). Can anyone imagine a method in which resource could be added to a server dynamically without it rebooting. That is as more players join the server, I can beef it up on the fly before I reach an upper limit and have to open a new island. I've considered loading a separate server instance then "transitioning" somehow all of the data from the small instance to the large instance but I just can't envisage a solution in which this could happen with no disruption from the players. Another alternative is somehow to run parts of the game simulation on different machines as I believe "professional" MMOs do. I imagine this is a significant rework of the engine, and well beyond my means.
I think, for now, that summarizes my main concerns and questions. I'd love to hear some feedback on peoples thoughts about the infrastructure.
Any questions or comments would be really useful, and thanks for reading this far!
Rob Sandbach
indieSpring Games
About Me
I have a feeling this may be a long post! Over at indiespring we're planning our "big project". I come from 3-5 years of using torque technology including releasing our recent game Fluttr, helping on the Combat Starter Kit, working on an FPS conversion from a mod and generally prototyping and playing around in Torque as an avid hobbiest. At the scripting level I'm extremely confident and comfortable with TGB, TGE and what I've played with in TGEA. At the C++ level I am a good understanding of torque's methods, have implemented many resources, and make small changes myself. My day job is in digital marketing, so I also have a fairly good understanding of the web & basic server architecture. I have a novice understanding of PHP, apache and the linux commandline and an extremely basic SQL comprehension (INSERT, UPDATE and DELETE). I also have experience with basic visualization through VMWare. IN the development of this project I would have access to experience database architects, and C++ coders to help me with deployment. It's not just me! So that's where I am.
About the Idea
The game that I am planning stems from my personal interests, my vision/understanding of the market place and my experience and access to resources in my day job. I have identified (what I believe to be) a potential niche in a persistent world server which hosts an FPS type game. That is, there are many players playing traditional multiplayer FPS's on non-persistant round based servers of 2-64 players which can be self hosted. There is also a category of FPS's (namely Planetside and WWIIonline) which aim to allow 1000s of concurrent players to war in a persistent, or campaign based, world. I am looking to build somewhat of a blend of these categories. A game in which a players character is persistent, has "statistics" which are remembered between sessions, will appear where they log off and so forth. There would be "one" server which players connect to and play on.
About the Technology
I am planning on building the server on cloud based infrastructure. For the following passage I assume that this is Amazon EC2 + S3 (though this is not 100%). Each Torque instance or zone server runs an "island". As an island fills up and gets close to it's maximum player count torque liases with AWS and spawns a new instance of an island server, which fires up. Some players are then offered an in-game incentive to withdraw from the current island, and drop into the new island to try and fight the battle there. This is the "basic" idea from which I want to build a scalable infrastructure in which costs of running are directly correlated to the number of players. As more players join, we add more islands and the resources available to them. As players leave, we come back down to one. It would not however scale to 1000s of players in one seamless world - I'd like to be realistic in my approach as a small studio. The goal of this post is to understand what is a realistic expectation.
My proposed server infrastructure is below:
There will also be a separate server responsible for patching and update delivery. (Darn my rushing!).
All database queries would go through the "custom" level written in java/python or something similar. These would communicate at the socket level with torque clients, as well as with the database servers. It's apparant that the "database" in that diagram may require replication/load balancing of it's own, but I have folk from my web development team who can handle that, for now I'm happy to assume that I have the actual database scaling covered. I'm more worried about understanding the game side of things.
The Questions
This thread is really to ask questions and gather advice on the technological and infrastructural implications of the game. Although I'd love to hear your feedback about the business model and player's opinions of such a game, if you could email them to me that would be fantastic. Here I'm really interested in answering the following, quite indepth, queries:
1.My first objective/priority is to get a T3D dedicated server instance on a given server running reliably with as many client as it reasonably can. I have read many threads on this and will be implementing the following changes. Tweaks to the networking parameters. Rewrite of the player class, possibly even the shapebase class to make it as lightweight as possible. Removing the sending of datablocks at mission load. Changing most rapid fire weapons to use raycats, not projectiles. Disabling polysoup collision and relying on the old conves collision hulls. No "dynamic physics" like destruction etc. To my questions. a) Are there any other obvious changes I've over looked that could be implemented and b) Aare DIFs my best bet for buildings/infrastructure. Will I get a significant performance gain from disaling poly soup and using them rather than traditional poly-based buildings?
2. In my scenario, is CPU likely to be the limiting factor, or memory? See [url
here for details on what specs an extra large sever instance will be. Is the "CPU Intensive" package going to significantly boost my player counts?
3. I know it's difficult/hard/impossible? to do so, but any rough indication or estimation of approximate player counts following these changes would be really useful to me.
4. When I prototype a T3D server on the instance, what's the best method to test the player count? Short of finding 100s of testers!?
5. Once I can get as many players as I can onto each island, I want to investigate, in the long run, making the transition between small instance, to a large instance on the same island server. At present this requires a reboot on Amazon's behalf (hence at least initially, each server will be running as an extra large instance server). Can anyone imagine a method in which resource could be added to a server dynamically without it rebooting. That is as more players join the server, I can beef it up on the fly before I reach an upper limit and have to open a new island. I've considered loading a separate server instance then "transitioning" somehow all of the data from the small instance to the large instance but I just can't envisage a solution in which this could happen with no disruption from the players. Another alternative is somehow to run parts of the game simulation on different machines as I believe "professional" MMOs do. I imagine this is a significant rework of the engine, and well beyond my means.
I think, for now, that summarizes my main concerns and questions. I'd love to hear some feedback on peoples thoughts about the infrastructure.
Any questions or comments would be really useful, and thanks for reading this far!
Rob Sandbach
indieSpring Games
#2
2) Good question. I would assume that a combination of CPU power and using raycasts instead of projectiles would move the bottleneck to some other issue like some physics check or collisions.
3) It would be near impossible for anyone to say without making those changes and testing. Server specs are always very fuzzy until the server is stress-tested.
4) I had heard of people modifying a client to strip out graphics so they could run multiple instances on multiple test machines, simulating many connections. Then, they ran bots through these clients, and you can probably use the AI scripts/code to do this and simulate players attacking each other and running around. How the stripped-down clients were done in specific was never mentioned, to my knowledge.
5) From a virtualization/cloud standpoint, I don't know. From a Torque standpoint, I don't know if it would be able to "see" or handle the change in resources going on under it. From a zoning standpoint, you can run multiple island instances and have a load-balancing master server shunt players to the lightest servers (also combining that with a player matching system like CoD4 has, possibly) on the fly, keeping a tally of how many players each instance has.
I realize that's not a lot of information, but for what it's worth, I hope it helped at least a bit...
09/30/2009 (9:51 pm)
1) Very smart moves towards the networking side of things. As for a) I think you can still use DTS/Collada buildings, using collision meshes for them that are less complex, as well, but testing is always king there.2) Good question. I would assume that a combination of CPU power and using raycasts instead of projectiles would move the bottleneck to some other issue like some physics check or collisions.
3) It would be near impossible for anyone to say without making those changes and testing. Server specs are always very fuzzy until the server is stress-tested.
4) I had heard of people modifying a client to strip out graphics so they could run multiple instances on multiple test machines, simulating many connections. Then, they ran bots through these clients, and you can probably use the AI scripts/code to do this and simulate players attacking each other and running around. How the stripped-down clients were done in specific was never mentioned, to my knowledge.
5) From a virtualization/cloud standpoint, I don't know. From a Torque standpoint, I don't know if it would be able to "see" or handle the change in resources going on under it. From a zoning standpoint, you can run multiple island instances and have a load-balancing master server shunt players to the lightest servers (also combining that with a player matching system like CoD4 has, possibly) on the fly, keeping a tally of how many players each instance has.
I realize that's not a lot of information, but for what it's worth, I hope it helped at least a bit...
#3
Is there any (communtity) work going on to get a Linux dedicated server working?
In DB terms I have read a lot about PostGres being better than MySQL for this sort of work due to the inclusion of "transactions". Can anyone explain that benefit for me?
10/02/2009 (11:06 am)
Thanks Ted, confirmation is just as valid and useful to me as suggestions!Is there any (communtity) work going on to get a Linux dedicated server working?
In DB terms I have read a lot about PostGres being better than MySQL for this sort of work due to the inclusion of "transactions". Can anyone explain that benefit for me?
#4
For your uses though, I don't think you should be overly worried. MySQL should be able to bear the brunt of your quieries nicely as long as the DB is set up properly. Most of your gameplay would be using physics or game logic rather than calls to the database to resolve issues like an RPG would, and the persistence part of it probably isn't as much overhead as that. So I myself would say go with what you know in this case. Of course, if testing shows otherwise, then by all means make the change, but I don't think you'll be clobbering the database too much.
10/02/2009 (1:15 pm)
Well, MySQL used to not have support for transactions as I'm told (I haven't mucked with it long enough to have known it then), but it does now, so that argument would be moot except on performance grounds, however they would differ.For your uses though, I don't think you should be overly worried. MySQL should be able to bear the brunt of your quieries nicely as long as the DB is set up properly. Most of your gameplay would be using physics or game logic rather than calls to the database to resolve issues like an RPG would, and the persistence part of it probably isn't as much overhead as that. So I myself would say go with what you know in this case. Of course, if testing shows otherwise, then by all means make the change, but I don't think you'll be clobbering the database too much.
Torque 3D Owner Rob Sandbach
indiespring
I should also add that in an ideal world, I would love to have 500 players on a server. If you need a benchmark of expectation I think that would be it.