How suitable IS TSE for an MMOG?
by Ian Winter · in Torque Game Engine Advanced · 06/22/2005 (5:46 am) · 53 replies
Ok apologies if I've missed this posted elsewhere, as if the gamedev.net forums are anything to go by every kid and his dog wants to produce an MMOG, but this is a more serious request - a basic feasibility study I suppose.
Essentially I'm wondering, now that Torque does support massive terrains, how feasible is an MMOG in Torque, something like say, Dark Age of Camelot or Planetside? What kind of work needs to be done on the engine level to support say 500, 1000, 3000 or 5000 clients. I'll admit I don't know an awful lot about Torque I've not used it an awful lot yet but I often see comments about Torque's networking code being scalable, the question I have is, how scalable?
I was always under the impression if you wanted to support more than a couple of hundred clients you'd need to start looking into clustering, but recent research I've done has led me to quotes stating that even a modern high-end desktop PC should be able to hand 1000 clients nowadays if the server is well written. I'd imagine you'd need to dumb down some of the physics, collision and combat calculations to acheive this but would something like this really be feasible in Torque whilst still keeping it a real-time enviroment?
If it was required to start using multiple machines to support 1000+ clients, how easy would it be to adapt Torque to handle this? Or would it simply be a better idea to start from scratch and write some custom server code but of course borrowing required bits (such as terrain calculations) from the TSE codebase? - I feel I'm competent enough to do this but I'm not sure if the extra work would be worth it if Torque can already do it perfectly well.
One final note is that I do realise that I'm going to need say, an SQL database at the backend to handle persistent objects and stuff - again, I can do things like that ok. I'm just unsure of what other work would be needed, I guess the biggest issue being setting Torque up to handle that number of clients.
Apologies if I've missed threads on this elsewhere, I'd be suprised if I haven't I think an MMOG is most developers dream to produce. I'd just like to get some hard facts to decide whether a decent MMOG project is feasible or not and get a good idea of what would be involved.
Thanks in advance!
Essentially I'm wondering, now that Torque does support massive terrains, how feasible is an MMOG in Torque, something like say, Dark Age of Camelot or Planetside? What kind of work needs to be done on the engine level to support say 500, 1000, 3000 or 5000 clients. I'll admit I don't know an awful lot about Torque I've not used it an awful lot yet but I often see comments about Torque's networking code being scalable, the question I have is, how scalable?
I was always under the impression if you wanted to support more than a couple of hundred clients you'd need to start looking into clustering, but recent research I've done has led me to quotes stating that even a modern high-end desktop PC should be able to hand 1000 clients nowadays if the server is well written. I'd imagine you'd need to dumb down some of the physics, collision and combat calculations to acheive this but would something like this really be feasible in Torque whilst still keeping it a real-time enviroment?
If it was required to start using multiple machines to support 1000+ clients, how easy would it be to adapt Torque to handle this? Or would it simply be a better idea to start from scratch and write some custom server code but of course borrowing required bits (such as terrain calculations) from the TSE codebase? - I feel I'm competent enough to do this but I'm not sure if the extra work would be worth it if Torque can already do it perfectly well.
One final note is that I do realise that I'm going to need say, an SQL database at the backend to handle persistent objects and stuff - again, I can do things like that ok. I'm just unsure of what other work would be needed, I guess the biggest issue being setting Torque up to handle that number of clients.
Apologies if I've missed threads on this elsewhere, I'd be suprised if I haven't I think an MMOG is most developers dream to produce. I'd just like to get some hard facts to decide whether a decent MMOG project is feasible or not and get a good idea of what would be involved.
Thanks in advance!
#42
That server CAN handle up to 3,000 players if the game server coded in the right way. We are talking about MMPRPG here, not MMPFPS. I'm not trying to look smart here ;) I just know what I'm talking about. I've seen games that barely handle 600 players on such machine...
And I can have 2000 simulated players on my dev machine at home (P4, 1Gb, 1Gb LAN network card) and it runs pretty in realtime :P
06/27/2005 (8:25 am)
@StefanThat server CAN handle up to 3,000 players if the game server coded in the right way. We are talking about MMPRPG here, not MMPFPS. I'm not trying to look smart here ;) I just know what I'm talking about. I've seen games that barely handle 600 players on such machine...
And I can have 2000 simulated players on my dev machine at home (P4, 1Gb, 1Gb LAN network card) and it runs pretty in realtime :P
#43
Did you setup 2k "virtual clients" on different machine(s) and have them all connect to the server using actual networking, or did you emulate them in the same process as the server?
The main reason I ask is that with "local" (same machine) connections, Torque automatically detects this and short circuits some of the more performance expensive network connections, so running them in the same process isn't going to be quite as accurate as having "real" network connections from external machines.
For what it's worth, I agree with you: while Torque is certainly not optimized in ANY way for this number of connections, that's because none of the games or demos made with any of the Torque apps had that number of clients intended as a design requirement. With proper re-design of the various things that, for example, iterate over the entire ClientGroup for example, Torque netcode can handle quite a bit more than has been "proven" thus far.
06/27/2005 (8:56 am)
Out of curiosity, how exactly did you simulate 2k players? Did you setup 2k "virtual clients" on different machine(s) and have them all connect to the server using actual networking, or did you emulate them in the same process as the server?
The main reason I ask is that with "local" (same machine) connections, Torque automatically detects this and short circuits some of the more performance expensive network connections, so running them in the same process isn't going to be quite as accurate as having "real" network connections from external machines.
For what it's worth, I agree with you: while Torque is certainly not optimized in ANY way for this number of connections, that's because none of the games or demos made with any of the Torque apps had that number of clients intended as a design requirement. With proper re-design of the various things that, for example, iterate over the entire ClientGroup for example, Torque netcode can handle quite a bit more than has been "proven" thus far.
#44
-Matt
P.S. Why does everyone like World of Warcraft so much? Not here, but just everywhere. I don't get it. It's a pretty average MMO.
06/27/2005 (9:34 am)
@Stefan Lundmark hmm...who to believe? You or Ben Garney? I'm gonna go with an employee, however I do agree with Stefan in some ways. First off, I seriously doubt you'd ever find enough players to fill up an entire server for an indie game. No offense to any indies, I've just never seen a game full with 3-4k players at the max. If sold commercially, you'd need a lot of servers. Also, if it were full to the max I think it would be laggy like hell. And when that happens the framerate will probably go to hell. An ORPG I was working on had this island that was completely lit on fire that had a horrible framerate offline that you had to look at the sky to even more moderately slowly. I suppose if you had a pretty good computer and a damn good internet connection, you could probably pull it off. Just my look at things...-Matt
P.S. Why does everyone like World of Warcraft so much? Not here, but just everywhere. I don't get it. It's a pretty average MMO.
#45
Sure, but TSE is not a MMPRPG. You said:
The stuff in bold is what made me react. If you modify it so heavily, then you can't call it a "TSE server" anymore, because it ain't.
Original TGE/TSE is not made to host such large numbers, like Stephen said above. And that's why I reacted on your comment.
@Stephen
I actually didn't know this at all. Is the difference above 10 percent?
06/27/2005 (9:48 am)
@AlexanderSure, but TSE is not a MMPRPG. You said:
Quote:
1) On a $1000/month value server (Xeon x2, 4Gb RAM, RAID, etc) a TSE server will host no more then 3000-4000 players. That's a fact. You will need to chunk the world into continents by servers to have 5000+ subscribers. No big olimpics with 100,000 viewers, sorry guys :)
The stuff in bold is what made me react. If you modify it so heavily, then you can't call it a "TSE server" anymore, because it ain't.
Original TGE/TSE is not made to host such large numbers, like Stephen said above. And that's why I reacted on your comment.
@Stephen
Quote:
The main reason I ask is that with "local" (same machine) connections, Torque automatically detects this and short circuits some of the more performance expensive network connections, so running them in the same process isn't going to be quite as accurate as having "real" network connections from external machines.
I actually didn't know this at all. Is the difference above 10 percent?
#46
I use the same trick as for my daily job (automated testing for databases and Web) - a simple application that repeats saved user actions (mouse, keyboard). I spawn them on 2-3 computers (not on the server) and they run for several hours, so we can spot the weak places in the database design and/or web application.
@Matt
You do can find players for your indie MMOG, look at Runestone (?).
People like WoW just because it has solid design, feeling and it's robust.
06/27/2005 (9:56 am)
@Stephen ZeppI use the same trick as for my daily job (automated testing for databases and Web) - a simple application that repeats saved user actions (mouse, keyboard). I spawn them on 2-3 computers (not on the server) and they run for several hours, so we can spot the weak places in the database design and/or web application.
@Matt
You do can find players for your indie MMOG, look at Runestone (?).
People like WoW just because it has solid design, feeling and it's robust.
#47
06/27/2005 (9:59 am)
Yeah, I'm not making one though. Might make one after my current project, but not in place of.
#48
No problem, it didn't hurt my feelings at all.
Just read the WHOLE sentence, not a part of it ;) The main idea was: it's not the constrain of the TSE server part, but the hardware (CPU, RAM), and you cannot "stretch" it over several computers. Even after the modification, you may call it TSE. A good game WILL modify sources of the engine, we have examples of it on GG.
The fact is, TSE is not prepeared for farming out of box, but if you marry it to the MPI ("Message Passing Interface", http://www-unix.mcs.anl.gov/mpi/ or http://www.lam-mpi.org/), you can create an unlimited virtual world, placing all the players into one continous world, creating olimpics with 100,000+ viewers ;), etc.
06/27/2005 (10:08 am)
@Stefan No problem, it didn't hurt my feelings at all.
Just read the WHOLE sentence, not a part of it ;) The main idea was: it's not the constrain of the TSE server part, but the hardware (CPU, RAM), and you cannot "stretch" it over several computers. Even after the modification, you may call it TSE. A good game WILL modify sources of the engine, we have examples of it on GG.
The fact is, TSE is not prepeared for farming out of box, but if you marry it to the MPI ("Message Passing Interface", http://www-unix.mcs.anl.gov/mpi/ or http://www.lam-mpi.org/), you can create an unlimited virtual world, placing all the players into one continous world, creating olimpics with 100,000+ viewers ;), etc.
#49
06/27/2005 (10:45 am)
Aww! Now I really want to make a MMO! This would be insane if someone made one with Torque and sold it in retail stores! Just think! You could make Lineage 2 or Guild Wars with Torque!!!
#50
Whilst a lot of new developers misunderstand the sheer amount of work in creating assets and such for an MMOG, there are some success stories in the MMOG world where a truly motivated and skilled individual or team who knows what they're in for and what to expect can acheive their goal.
Certainly an indie MMOG isn't something that's as easy to produce and as much of a common happening as the likes of a single player indie game whilst it's still an extremely difficult and time consuming task it's certainly not been proven to be in the realm of impossible ;) I'm all for encouraging indies to look at making an MMOG personally but I do agree it needs a lot of proper research which is what I'm mostly trying to do now, I'd like to think at least if I fail in the task I can eventually create a site of my experiences, what went wrong and so on so that other indie teams researching it can see if they can bypass the hurdles I stumbled and fell on.
06/27/2005 (2:54 pm)
Without going too off topic, IIRC one of the most popular Asian MMOGs that has one of the biggest MMOG playerbases in the world was created entirely by one or two people as an indie project.Whilst a lot of new developers misunderstand the sheer amount of work in creating assets and such for an MMOG, there are some success stories in the MMOG world where a truly motivated and skilled individual or team who knows what they're in for and what to expect can acheive their goal.
Certainly an indie MMOG isn't something that's as easy to produce and as much of a common happening as the likes of a single player indie game whilst it's still an extremely difficult and time consuming task it's certainly not been proven to be in the realm of impossible ;) I'm all for encouraging indies to look at making an MMOG personally but I do agree it needs a lot of proper research which is what I'm mostly trying to do now, I'd like to think at least if I fail in the task I can eventually create a site of my experiences, what went wrong and so on so that other indie teams researching it can see if they can bypass the hurdles I stumbled and fell on.
#51
That would be an even larger project than a indie MMORPG alone :p
06/27/2005 (3:02 pm)
Quote:
The fact is, TSE is not prepeared for farming out of box, but if you marry it to the MPI ("Message Passing Interface", http://www-unix.mcs.anl.gov/mpi/ or http://www.lam-mpi.org/), you can create an unlimited virtual world, placing all the players into one continous world, creating olimpics with 100,000+ viewers ;), etc.
That would be an even larger project than a indie MMORPG alone :p
#52
You'd want to create a custom database server that would handle communicating with your database. You wouldn't want the overhead of Torque on that, that's best as an external app. Likewise the login server/patch server/chat servers probably shouldn't be using TSE. Not saying that you couldn't, but you don't need 90% of the features of Torque for these applications.
You'd want to create a server application which would be responsible for managing the other servers, especially if you were trying to create a seamless world, or one with instances. This application would be responsible to check to see if servers are up at all times, for creating instances, for re-routing in case of a crash, for switching to backup servers in the instance that your database server/login servers crash, etc. Once again, this probably shouldn't be using TSE.
The world servers/zone servers are where Torque should be used, and they would constitute the bulk of your game. All the other stuff is transparent to clients, with the exception of the login server. Torque with some small modifications is more than capable of handling this.
06/27/2005 (9:10 pm)
Torque would work well for the world/zone servers. You would have to create some additional servers though to handle various operations for your MMOG that Torque isn't going to do well out of the box. I'd personally recommend not even using Torque for those operations, and writing a custom solution for them. If you use Torque, the TNL might be better than TSE for those. For the bulk of the game though TSE would work fine.You'd want to create a custom database server that would handle communicating with your database. You wouldn't want the overhead of Torque on that, that's best as an external app. Likewise the login server/patch server/chat servers probably shouldn't be using TSE. Not saying that you couldn't, but you don't need 90% of the features of Torque for these applications.
You'd want to create a server application which would be responsible for managing the other servers, especially if you were trying to create a seamless world, or one with instances. This application would be responsible to check to see if servers are up at all times, for creating instances, for re-routing in case of a crash, for switching to backup servers in the instance that your database server/login servers crash, etc. Once again, this probably shouldn't be using TSE.
The world servers/zone servers are where Torque should be used, and they would constitute the bulk of your game. All the other stuff is transparent to clients, with the exception of the login server. Torque with some small modifications is more than capable of handling this.
Torque Owner Bryce "Cogburn" Weiner
My one true gaming claim to fame was an extremely successfull web-based game.
*Over 4 million records in 42 tables in 3 databases
*263,293 lines of code
*25 teams, that were populated by 72 units over the course of 2 years.
*2,782 players
*124MB of disk space (including databases)
*182 dynamic HTML pages backed by PHP
*27 Flash applets
*5 programming languages (HTML, PHP, JavaScript, ActionScript, SQL)
*Over 2,000 hours of coding, art design and maintenance
*Over 1,000,000 hits per month, ~250,000 unique
The whole thing was backed w/ mySQL. On a CHEAP system (P2 or thereabouts), mySQL is an absolute dog. The lack of transactions and its use of table locking completely lags output, and does so exponentially. Once a significant queue of requests is created, the server will continue to slow the more requests are entered into the queue (and thereby forcing the server to swap out memory to disk, and on and on...).
Now that was on my CHEAP server....
On my nice dual P3 700Mhz server... wow. With a high load (> 1,000 unique requests per minute) there was not enough output slowdown that a user would notice... or even me as a developer. Request times were < 0.035 seconds for complex INNER JOINs (joins using more than 4 tables).
It must also be said, my code went through 4 iterations of optimization. Its also worth saying that mySQL ran on the SAME SERVER as Apache, so it was ALSO handling all those html requests too. There is a good reason why people use mySQL even though it isn't the most advanced out there. It's light, it's free, and it's DAMN good for what you pay for it.
Think about the number of transactions for a moment.... if your MMO requires > 1,000 DATABASE requests per minute a) you need to optimize your code or b) you are so successfull you should already be charging your players.
SHAMELESS PLUG:
I setup a small shrine to it here