How suitable IS TSE for an MMOG?
by Ian Winter · in Torque Game Engine Advanced · 06/22/2005 (5:46 am) · 53 replies
Ok apologies if I've missed this posted elsewhere, as if the gamedev.net forums are anything to go by every kid and his dog wants to produce an MMOG, but this is a more serious request - a basic feasibility study I suppose.
Essentially I'm wondering, now that Torque does support massive terrains, how feasible is an MMOG in Torque, something like say, Dark Age of Camelot or Planetside? What kind of work needs to be done on the engine level to support say 500, 1000, 3000 or 5000 clients. I'll admit I don't know an awful lot about Torque I've not used it an awful lot yet but I often see comments about Torque's networking code being scalable, the question I have is, how scalable?
I was always under the impression if you wanted to support more than a couple of hundred clients you'd need to start looking into clustering, but recent research I've done has led me to quotes stating that even a modern high-end desktop PC should be able to hand 1000 clients nowadays if the server is well written. I'd imagine you'd need to dumb down some of the physics, collision and combat calculations to acheive this but would something like this really be feasible in Torque whilst still keeping it a real-time enviroment?
If it was required to start using multiple machines to support 1000+ clients, how easy would it be to adapt Torque to handle this? Or would it simply be a better idea to start from scratch and write some custom server code but of course borrowing required bits (such as terrain calculations) from the TSE codebase? - I feel I'm competent enough to do this but I'm not sure if the extra work would be worth it if Torque can already do it perfectly well.
One final note is that I do realise that I'm going to need say, an SQL database at the backend to handle persistent objects and stuff - again, I can do things like that ok. I'm just unsure of what other work would be needed, I guess the biggest issue being setting Torque up to handle that number of clients.
Apologies if I've missed threads on this elsewhere, I'd be suprised if I haven't I think an MMOG is most developers dream to produce. I'd just like to get some hard facts to decide whether a decent MMOG project is feasible or not and get a good idea of what would be involved.
Thanks in advance!
Essentially I'm wondering, now that Torque does support massive terrains, how feasible is an MMOG in Torque, something like say, Dark Age of Camelot or Planetside? What kind of work needs to be done on the engine level to support say 500, 1000, 3000 or 5000 clients. I'll admit I don't know an awful lot about Torque I've not used it an awful lot yet but I often see comments about Torque's networking code being scalable, the question I have is, how scalable?
I was always under the impression if you wanted to support more than a couple of hundred clients you'd need to start looking into clustering, but recent research I've done has led me to quotes stating that even a modern high-end desktop PC should be able to hand 1000 clients nowadays if the server is well written. I'd imagine you'd need to dumb down some of the physics, collision and combat calculations to acheive this but would something like this really be feasible in Torque whilst still keeping it a real-time enviroment?
If it was required to start using multiple machines to support 1000+ clients, how easy would it be to adapt Torque to handle this? Or would it simply be a better idea to start from scratch and write some custom server code but of course borrowing required bits (such as terrain calculations) from the TSE codebase? - I feel I'm competent enough to do this but I'm not sure if the extra work would be worth it if Torque can already do it perfectly well.
One final note is that I do realise that I'm going to need say, an SQL database at the backend to handle persistent objects and stuff - again, I can do things like that ok. I'm just unsure of what other work would be needed, I guess the biggest issue being setting Torque up to handle that number of clients.
Apologies if I've missed threads on this elsewhere, I'd be suprised if I haven't I think an MMOG is most developers dream to produce. I'd just like to get some hard facts to decide whether a decent MMOG project is feasible or not and get a good idea of what would be involved.
Thanks in advance!
#2
About Databases: ODBC and/or libMySQL both have implementations for TGE that can easily be ported over to TSE... It's so easy to port them its the reason why there are no resources covering it. Some basic programming knowledge is all that is required. Search the Resources section for ODBC or libmysql and you'll see them. If that's not your taste, there is an XML persistance engine for TGE that could be ported over with a little more work (for a games like TES).
About Terrain: At the risk of sounding trite, the Atlas Terrain Engine in TSE makes the size and detail of PlanetSide maps look like Tribes 2 mods. Assuming you could provide a server that could handle it, you could run all 10 continents of PlanetSide AS ONE CONTIGUOUS WORLD MAP. The engine can ABSOLUTELY handle a TO SCALE planetary terrain without issue. I've never played DOAC, but I am an avid PlanetSider... If PS had the Atlas engine I can't imagine the kind of game it could have been! As Ben Garney put it:
THAT is terrain management done right. ;)
About MMOs and TSE/TGE: With TSE (and the formal completion of the TNL), the Torque system is TECHNICALLY able to handle the demands of an MMO, BUT there is a LOT of optimization that you will need to do for your own game. There are no TECHINCAL ENGINE barriers to the number of players that a server can handle. There is, however, the practical element of requiring a $250k BladeServer to run that single world and all of its inhabitants. Sure, the software CAN TECHNICALLY do it, but you'll need some serious load balancing server farms to make it playable with large numbers of players.
Summation: IMHO, TSE is (almost, pending release) the OPTIMAL platform for the development of a Proof of Concept of an MMO. You can ABSOLUTELY make a game visually stunning enough and deep enough to at least give the atmosphere and feel of your idea. It may take a LOT of work to get it to a point ready for distribution, but to get your ideas across to the Big Money People you may be pitching your idea to, there is nothing better. Again, IMHO, if you aren't super-rich with money to burn, making an indie MMO that you plan on marketing and operating yourself is a huge mistake. If you are hell-bent on making an MMO, start with the thought in mind that you are designing something to prove that an idea for a game is workable so that you can sell to the likes of SOE or ArenaNet.
Hope that helps answer some of your questions. :)
06/22/2005 (6:58 am)
I get posts deleted all the time for speaking out of turn... so maybe this one will stay up. :)About Databases: ODBC and/or libMySQL both have implementations for TGE that can easily be ported over to TSE... It's so easy to port them its the reason why there are no resources covering it. Some basic programming knowledge is all that is required. Search the Resources section for ODBC or libmysql and you'll see them. If that's not your taste, there is an XML persistance engine for TGE that could be ported over with a little more work (for a games like TES).
About Terrain: At the risk of sounding trite, the Atlas Terrain Engine in TSE makes the size and detail of PlanetSide maps look like Tribes 2 mods. Assuming you could provide a server that could handle it, you could run all 10 continents of PlanetSide AS ONE CONTIGUOUS WORLD MAP. The engine can ABSOLUTELY handle a TO SCALE planetary terrain without issue. I've never played DOAC, but I am an avid PlanetSider... If PS had the Atlas engine I can't imagine the kind of game it could have been! As Ben Garney put it:
Quote:
Atlas's current incarnation supports areas as large as 10^154 chunks on a side, for a grand total of 6.5 x 10 ^ 312 chunks on the lowest level of the quadtree. For reference, the surface area of the earth is approximately 5.1 x 10 ^ 15 square meters...
...
For reference, Pluto's mean distance from the sun is 5.9e12 meters. So for the theoretical maximum Atlas terrain (which would still render at something approximating realtime... theoretically ;), we can fit something like 10e145 SOLAR SYSTEMS in that working set.
...
It would be 2.64e141 lightyears on a side. You could fit approximately 1.8e128 copies of the visible universe along one side of this terrain, and you could run across it at a normal running speed and be impressed the whole way at the detailed nature of it.
THAT is terrain management done right. ;)
About MMOs and TSE/TGE: With TSE (and the formal completion of the TNL), the Torque system is TECHNICALLY able to handle the demands of an MMO, BUT there is a LOT of optimization that you will need to do for your own game. There are no TECHINCAL ENGINE barriers to the number of players that a server can handle. There is, however, the practical element of requiring a $250k BladeServer to run that single world and all of its inhabitants. Sure, the software CAN TECHNICALLY do it, but you'll need some serious load balancing server farms to make it playable with large numbers of players.
Summation: IMHO, TSE is (almost, pending release) the OPTIMAL platform for the development of a Proof of Concept of an MMO. You can ABSOLUTELY make a game visually stunning enough and deep enough to at least give the atmosphere and feel of your idea. It may take a LOT of work to get it to a point ready for distribution, but to get your ideas across to the Big Money People you may be pitching your idea to, there is nothing better. Again, IMHO, if you aren't super-rich with money to burn, making an indie MMO that you plan on marketing and operating yourself is a huge mistake. If you are hell-bent on making an MMO, start with the thought in mind that you are designing something to prove that an idea for a game is workable so that you can sell to the likes of SOE or ArenaNet.
Hope that helps answer some of your questions. :)
#3
06/22/2005 (6:59 am)
Oh and whilst I'm on the subject by the way, how does the new terrain work server side? Whilst I realise it pages in client side I assume it just needs to be able to load in the entire block server side? Or is it intelligent enough to only load in blocks where players are? I'm just wondering, if you had a 1gb set of terrain data you'll need over a gb of RAM in the server for example?
#4
@ Bryce, when you mention load balancing server farms - what rough number of players are you thinking about for that kind of hardware? baring in mind we only need extremely basic collision detection and such (i.e. no locational damage like headshots and stuff). I think one of the biggest issues I'm struggling with is the kind of hardware you need per-amount of players. As I say the comments I've seen from people in the industry even suggest you only need your average 3.5ghz or so desktop machine to handle 1000 clients. I'm trying to get a rough idea of how realistic that goal is (they might just be trying to show off in an "I'm so good at coding I can get a 1000 client FPS running on a desktop machine" kind of way, or they maybe right, that's what I'm trying to figure out) and at what point (in number of players) you need to start sharing out processing load to 2+ machines.
06/22/2005 (7:05 am)
Thanks for the replies so far guys ;)@ Bryce, when you mention load balancing server farms - what rough number of players are you thinking about for that kind of hardware? baring in mind we only need extremely basic collision detection and such (i.e. no locational damage like headshots and stuff). I think one of the biggest issues I'm struggling with is the kind of hardware you need per-amount of players. As I say the comments I've seen from people in the industry even suggest you only need your average 3.5ghz or so desktop machine to handle 1000 clients. I'm trying to get a rough idea of how realistic that goal is (they might just be trying to show off in an "I'm so good at coding I can get a 1000 client FPS running on a desktop machine" kind of way, or they maybe right, that's what I'm trying to figure out) and at what point (in number of players) you need to start sharing out processing load to 2+ machines.
#5
As far as specific load, your game will dictate that more than the limitations (or lack thereof) of TSE. If you are creating a 100% open, contiguous world for 1,000 players a ROUGH.... ROUGH.... ROUGH.... estimate would be one server per 256 players. This is not a TSE limitation, but simply an educated guess based on the performance of the TGE. I consider one server would be your standard dual P42.2Ghz Xeon that you can pickup bargain basement for around 2k. This does NOT take into account bandwidth limitations, but does provide the amount of computing power to so that your players are at least not waiting on the server. Multiple processor systems are always superior to single processor systems because of the parallel processing, even if the CPU's themselves run slower the output is exponentially faster.
Also, you will need additional servers for login management and of course the databases.
Keep in mind, during development a PIII 733MHz to handle everything would do just as well. I have just that setup for my team of 5 and it does us just fine... for now.
Short answer: TSE can do what you want, in theory. In practice, you will need to study the load created by the various common player operations based on YOUR code and scale up from there.
06/22/2005 (1:27 pm)
I exaggerate to clarify. :) The "Wish" (failed MMO) servers were just auctioned off on eBay for $120,000. They estimated an initial player base of 68,000. A joke in poor taste, perhaps.As far as specific load, your game will dictate that more than the limitations (or lack thereof) of TSE. If you are creating a 100% open, contiguous world for 1,000 players a ROUGH.... ROUGH.... ROUGH.... estimate would be one server per 256 players. This is not a TSE limitation, but simply an educated guess based on the performance of the TGE. I consider one server would be your standard dual P42.2Ghz Xeon that you can pickup bargain basement for around 2k. This does NOT take into account bandwidth limitations, but does provide the amount of computing power to so that your players are at least not waiting on the server. Multiple processor systems are always superior to single processor systems because of the parallel processing, even if the CPU's themselves run slower the output is exponentially faster.
Also, you will need additional servers for login management and of course the databases.
Keep in mind, during development a PIII 733MHz to handle everything would do just as well. I have just that setup for my team of 5 and it does us just fine... for now.
Short answer: TSE can do what you want, in theory. In practice, you will need to study the load created by the various common player operations based on YOUR code and scale up from there.
#6
One of the things you will want to consider is that Tribes 2 lived in a world where sub-second timing was critical with regard to player-input, physics, and network synchronization. An MMO does not live in that world. Turning down just the tick frequency would probably increase, greatly, the number of players you can get on a server.
06/22/2005 (2:01 pm)
We had an MMO team working on Torque, but due to an unfortunat series of events the project no longer exists. They were able to have a lot more than 256 clients on one server running Torque but I'm not sure of all the details of what they changed to do so.One of the things you will want to consider is that Tribes 2 lived in a world where sub-second timing was critical with regard to player-input, physics, and network synchronization. An MMO does not live in that world. Turning down just the tick frequency would probably increase, greatly, the number of players you can get on a server.
#7
I've got a few boxes here, a 1.8ghz 1gb RAM machine, a normal 2.8ghz Windows box that I could use and a 1.2ghz Linux box (this'll probably do the database stuff). I've then also got a 2ghz laptop, 2.4ghz laptop and a 3.2ghz desktop all on a 1gb LAN so I have plenty of resources to work up a decent kind of test rig and hopefully spread across my non-server machines I can emulate a high amount of clients.
I guess from the info here it sounds perfectly feasible so I'll give it a go, probably knock together some kind of test zone then see if I can produce some kind of setup to emulate multiple clients, of course, staying in the spirit of Torque I'll certainly post any results and useful info I come up with ;)
06/22/2005 (2:32 pm)
Yeah, spells/weapons will be cast/swung only once every couple of seconds, wont have to deal with machine guns calculating 100 rounds per minute or whatever so that should help a lot ;)I've got a few boxes here, a 1.8ghz 1gb RAM machine, a normal 2.8ghz Windows box that I could use and a 1.2ghz Linux box (this'll probably do the database stuff). I've then also got a 2ghz laptop, 2.4ghz laptop and a 3.2ghz desktop all on a 1gb LAN so I have plenty of resources to work up a decent kind of test rig and hopefully spread across my non-server machines I can emulate a high amount of clients.
I guess from the info here it sounds perfectly feasible so I'll give it a go, probably knock together some kind of test zone then see if I can produce some kind of setup to emulate multiple clients, of course, staying in the spirit of Torque I'll certainly post any results and useful info I come up with ;)
#8
06/22/2005 (2:45 pm)
You might want to look at the RTS kit for an idea of what you can accomplish using trimmed down networking. 
#9
-Josh Ritter
Prairie Games
06/22/2005 (3:13 pm)
I've been meaning to drop the tick frequency... I am going to give this a shot and see what artifacts are introduced...-Josh Ritter
Prairie Games
#10
I always figured an MMORPG has a LOT of scripting going on in the background managing game events that the user is never aware of. RPG inventories are always more complex than the likes of Tribes, so you figure that overhead is tossed in there too. It's going to be substantially more than Tribes, either way you slice it.
The cost of dual-proc servers are so low now that when anyone mentions purchasing a PC for server-type use I always say dual-proc first. :)
06/22/2005 (7:52 pm)
Again, it was a rough estimate of players per physical server.I always figured an MMORPG has a LOT of scripting going on in the background managing game events that the user is never aware of. RPG inventories are always more complex than the likes of Tribes, so you figure that overhead is tossed in there too. It's going to be substantially more than Tribes, either way you slice it.
The cost of dual-proc servers are so low now that when anyone mentions purchasing a PC for server-type use I always say dual-proc first. :)
#11
Is it right to assume this code change with halve the network update rate?
If so, after a few small fixups... seems to work great!
-Josh Ritter
Prairie Games
06/22/2005 (9:39 pm)
#define TickShift 6 // <---- WAS 5!!!! #define TickMs (1 << TickShift) #define TickSec (F32(TickMs) / 1000.0f) #define TickMask (TickMs - 1)
Is it right to assume this code change with halve the network update rate?
If so, after a few small fixups... seems to work great!
-Josh Ritter
Prairie Games
#12
06/22/2005 (10:36 pm)
Josh, this should reduce the computation for physics, networking, almost everything by half. I could be totally wrong, and would admit it if so because I don't know for sure *everything* that runs off of that tick frequency, however anything that is derived from GameBase, and uses ticks to do processing will be halved.
#13
06/22/2005 (10:50 pm)
@Josh: I think the network packet update and the processing ticks are different values, there's a variable somewhere that sets the network data rate, it's been a while so I might be confusing that with this, but you might want to check to be sure!
#14
I'll have to play with these for our next patch...
-Josh Ritter
Prairie Games
06/22/2005 (10:55 pm)
$Pref::Net::LagThreshold = "400"; $pref::Net::PacketRateToClient = "10"; $pref::Net::PacketRateToServer = "32"; $pref::Net::PacketSize = "200";
I'll have to play with these for our next patch...
-Josh Ritter
Prairie Games
#15
06/22/2005 (11:01 pm)
That's what I was talking about! :)
#16
I was still talking about the Tick rate, though, that will substantially reduce CPU load. Tweeking the packet stuff, though, will help the net traffic for sure.
06/23/2005 (11:48 am)
Oh :-PI was still talking about the Tick rate, though, that will substantially reduce CPU load. Tweeking the packet stuff, though, will help the net traffic for sure.
#17
06/23/2005 (2:51 pm)
Yeah the tick rate should help a lot with CPU, although, I wonder if it will not mess with vehicle physics/integration? That depends on the tick time too...
#18
Need to look into the packet stuff soon and see what we can get away with...
-Josh Ritter
Prairie Games
06/23/2005 (2:55 pm)
So far things have been fine with halving the tick rate... I should also maybe try sleeping the dedicated server process during this time.... we don't have any vehicles, I could see the tick rate affecting these adversely... Need to look into the packet stuff soon and see what we can get away with...
-Josh Ritter
Prairie Games
#19
It shouldn't mess *too* much with the physics, because it's still going to calculate the new solution every tick, it's just twice as long between ticks.
06/23/2005 (4:17 pm)
Xavier: It will mess with the physics stuff, however what I was saying was that Minions doesn't need that precise of a physics update freq. It shouldn't mess *too* much with the physics, because it's still going to calculate the new solution every tick, it's just twice as long between ticks.
#20
@Path: Sorry, that makes sense, if he doesn't need physics everything should be cool :)
06/23/2005 (4:58 pm)
@Josh: That's good to hear then.@Path: Sorry, that makes sense, if he doesn't need physics everything should be cool :)
Torque Owner Kevin Johnson
nope... that is actually quite easy..the biggest problem with building a MMOG is art assets and game play.
Check out chapter2 of Torque Game Engine Documentation Version 1.3.x "Creating a Successful Indie Game With Torque" there is alot of good info in there.. :)
goodluck