General Networking Discussion--Game Genres
by Stephen Zepp · in Torque Game Engine · 11/30/2008 (9:09 am) · 31 replies
(Note: this thread was originally comments posted in Tony Richard's recent Fractured Universe .plan, but it was bad form on my part to derail his .plan, so here's the comments related to Torque networking):
Stephen Zepp said:
With about 7 changes to the underlying code and a few adjustments to previously programmed parameters, you can easily serve 200 players with a decent network (not even a good one)--and that's more than Everquest serves even today per zone.
With a couple more months of well informed architectural decisions and implementation, you can serve Torque across distributed servers and have as many players as you wish up to your main server farm input pipe.
Stephan Bondier asked:
Sorry to hijack the discussion, but in this case, why isn't it set as a default in the distribution?
Marcus responded:
I think it is important to remember that your 1kb may not be very adequate for twitch based gameplay...say a MMOFPS.
Stephen Zepp responded:
Because Torque isn't marketed as an MMO back end architectural solution. Each game type and each game has different, and sometimes mutually exclusive needs. Torque provides examples for three different genres (FPS, Racing, RTS with a separate purchase) that each tune the networks for what is appropriate for that type of genre, but the underlying networking paradigm is the same--it's just used differently for each of the genres.
Stephen Zepp said:
With about 7 changes to the underlying code and a few adjustments to previously programmed parameters, you can easily serve 200 players with a decent network (not even a good one)--and that's more than Everquest serves even today per zone.
With a couple more months of well informed architectural decisions and implementation, you can serve Torque across distributed servers and have as many players as you wish up to your main server farm input pipe.
Stephan Bondier asked:
Quote:
With about 7 changes to the underlying code and a few adjustments to previously programmed parameters, you can easily serve 200 players with a decent network (not even a good one)--and that's more than Everquest serves even today per zone.
Sorry to hijack the discussion, but in this case, why isn't it set as a default in the distribution?
Marcus responded:
I think it is important to remember that your 1kb may not be very adequate for twitch based gameplay...say a MMOFPS.
Stephen Zepp responded:
Quote:
Sorry to hijack the discussion, but in this case, why isn't it set as a default in the distribution?
Because Torque isn't marketed as an MMO back end architectural solution. Each game type and each game has different, and sometimes mutually exclusive needs. Torque provides examples for three different genres (FPS, Racing, RTS with a separate purchase) that each tune the networks for what is appropriate for that type of genre, but the underlying networking paradigm is the same--it's just used differently for each of the genres.
#22
Using P2P for patching is different to using it for game content imho. A game hacker may have a list of IP's that play WoW, but they'll have no real way to tie those IP's to player accounts.
Where as with a p2p chat system, all it takes is talking to the wrong person at the wrong time and before you know it, you're on the end of a DOS just as your engaging in a PvP fight or have just finished off a huge boss and were ready to loot.
Putting aside the DOS concern. Isn't there still a real need to pass chat data through a company server in order to allow support staff to investigate cases of harassment?
12/19/2008 (5:17 pm)
Quote:Stephen Zepp: Peer to peer anything out in the wild is a risky solution, one that many commercial companies aren't too excited about--and many knowledgeable customers are afraid of as well.
Quote:Tony Richards: Correct me if I'm wrong, but doesn't WoW use it for distributing content updates, as do a whole lot of other commercial (non-game related) companies.
Using P2P for patching is different to using it for game content imho. A game hacker may have a list of IP's that play WoW, but they'll have no real way to tie those IP's to player accounts.
Where as with a p2p chat system, all it takes is talking to the wrong person at the wrong time and before you know it, you're on the end of a DOS just as your engaging in a PvP fight or have just finished off a huge boss and were ready to loot.
Putting aside the DOS concern. Isn't there still a real need to pass chat data through a company server in order to allow support staff to investigate cases of harassment?
#23
@Gary:
Needing to store a copy of the chat log was the main reason I implemented a chat server for my own project -- as well as allowing external IRC-like access. I was originally going to use Torque to relay local chat to the players, and just pass it back to my chat server for storage. But with the NetEvent priority point Stephen brought up, I'm going to move it completely out-of-band.
- Dave
12/19/2008 (9:39 pm)
Maybe we could move the what is an indie?, indies must innovate, and innovation != Torque discussion to another thread? Perhaps this thread should have been titled Torque General Networking Discussion, as it appears that is what Stephen was intending to discuss. And I'm still interested in hearing what he has to say based on his years of experience on the topic.@Gary:
Needing to store a copy of the chat log was the main reason I implemented a chat server for my own project -- as well as allowing external IRC-like access. I was originally going to use Torque to relay local chat to the players, and just pass it back to my chat server for storage. But with the NetEvent priority point Stephen brought up, I'm going to move it completely out-of-band.
- Dave
#24
12/20/2008 (5:53 am)
@David - Done... I'll post a link when I am done posting the new topic. 
#25
The reason of why i am investigating the better form of show html content in tge, is for use it to show online help, forums, chat, etc... that runs in other server.
www.garagegames.com/blogs/60581/15798
12/20/2008 (8:00 am)
MMO:The reason of why i am investigating the better form of show html content in tge, is for use it to show online help, forums, chat, etc... that runs in other server.
www.garagegames.com/blogs/60581/15798
#26
For the sake of getting a very interesting and valuable conversation on TGE/A network optimizations up and running again.
[/BUMP]
I think one thing that would benefit online games with heavy network loads (and possibly dynamic changes to the world) would be to find a way to use datablocks... without using datablocks. There had been talk of caching, which reduces the load time for them, but what would the network effects be of taking the datablock info and moving it over to the object-side of the network and having it update those only when it needs to?
Also, from my searching on this site, some of the things that should be looked at when looking at trimming network resources for your game:
1) Packet rates (both to client and to server). They default at 32 per second, at a size of 450, I believe. You can probably trim the rate back while bumping up the packet size a bit (not too far, or else you hit the hardware MTU, and then the packets start getting broken up for transit).
2) Someone had suggested slowing the tick rate in the game. I haven't played with this, and generally speaking, it is something you should consider carefully, as some of the reported side effects were spongy camera systems, collision problems (probably due to prediction), etc.
3) If it's an MMO, you could probably hard-code physics into the client and simply not transmit those values over the network, either on the object or datablock side. I haven't looked at that too closely yet myself, so I don't know what kind of savings that would give, but most MMOs don't rely heavily on physics, so it is a bit of wasted bandwidth.
4) Not packing and unpacking states and movement if they remain the same from the last update. Again, something I saw mentioned in other threads along these lines, but I haven't looked too closely into it myself yet.
Food for thought. Any other ideas, experiences, questions, or sage advice?
02/01/2009 (7:29 am)
[BUMP]For the sake of getting a very interesting and valuable conversation on TGE/A network optimizations up and running again.
[/BUMP]
I think one thing that would benefit online games with heavy network loads (and possibly dynamic changes to the world) would be to find a way to use datablocks... without using datablocks. There had been talk of caching, which reduces the load time for them, but what would the network effects be of taking the datablock info and moving it over to the object-side of the network and having it update those only when it needs to?
Also, from my searching on this site, some of the things that should be looked at when looking at trimming network resources for your game:
1) Packet rates (both to client and to server). They default at 32 per second, at a size of 450, I believe. You can probably trim the rate back while bumping up the packet size a bit (not too far, or else you hit the hardware MTU, and then the packets start getting broken up for transit).
2) Someone had suggested slowing the tick rate in the game. I haven't played with this, and generally speaking, it is something you should consider carefully, as some of the reported side effects were spongy camera systems, collision problems (probably due to prediction), etc.
3) If it's an MMO, you could probably hard-code physics into the client and simply not transmit those values over the network, either on the object or datablock side. I haven't looked at that too closely yet myself, so I don't know what kind of savings that would give, but most MMOs don't rely heavily on physics, so it is a bit of wasted bandwidth.
4) Not packing and unpacking states and movement if they remain the same from the last update. Again, something I saw mentioned in other threads along these lines, but I haven't looked too closely into it myself yet.
Food for thought. Any other ideas, experiences, questions, or sage advice?
#27
1. static emitters: Examples would be fire, waterfalls/fountains, things that don't have to be in sync with the server, and actually, the server doesn't even need to know about them eccept they exist and never move.
2. moveable emitters: footpuffs, tire dust, projectiles, etc... things that do move and need to be rendered in sync to all players.
I haven't delved into torques particles, so this may in fact be the way they are rendered, but I don't think so. As it stands now, if you have a large number of emitters, say fire, and the emitters are quite large, you will experience lag on the network, where it isn't noticeable in a single player game.
02/01/2009 (7:53 am)
Something I have been thinking about, but don't know if it would actually save anything, was to split up the particle emitter system. From my understanding, most, if not all the particles are rendered from the server. To me, this is a total waste of resources. The split would be:1. static emitters: Examples would be fire, waterfalls/fountains, things that don't have to be in sync with the server, and actually, the server doesn't even need to know about them eccept they exist and never move.
2. moveable emitters: footpuffs, tire dust, projectiles, etc... things that do move and need to be rendered in sync to all players.
I haven't delved into torques particles, so this may in fact be the way they are rendered, but I don't think so. As it stands now, if you have a large number of emitters, say fire, and the emitters are quite large, you will experience lag on the network, where it isn't noticeable in a single player game.
#28
As for packet rates, that's less important than scoping and the contents of those packets. I think those default values are targeting a modem running at 14.4k, but I could be wrong on the history there. I don't think the "default" has much relevance to reality these days.
If you're just looking at "tweaking" Torque to reduce bandwidth utilization then possibly Stephen Zepp will be able to provide you with more insight.
I tried for quite a long time and I couldn't come up with any good tweaks that had more than a 50% impact, but maybe that's good enough for your needs.... and it ended up being more than just a little "tweak".
The simple fix:
You can improve the "scope" logic and cache the datablocks, stop replicating the cameras, and improve the logic so that fewer bytes are sent per object (i.e. do your best to only send updates for objects that have changed and only exactly what changed). The RTS Starter Kit has good examples on "network light" objects.
Using these techniques I was able to reduce Torque's bandwidth utilization by about 50% of stock Torque.
But, if that's not good enough, (as it wasn't for my needs)....
(continued)
02/02/2009 (10:15 am)
@Ted - Heh using datablocks without using datablocks... In the context of MMO games, I know exactly what you mean.As for packet rates, that's less important than scoping and the contents of those packets. I think those default values are targeting a modem running at 14.4k, but I could be wrong on the history there. I don't think the "default" has much relevance to reality these days.
If you're just looking at "tweaking" Torque to reduce bandwidth utilization then possibly Stephen Zepp will be able to provide you with more insight.
I tried for quite a long time and I couldn't come up with any good tweaks that had more than a 50% impact, but maybe that's good enough for your needs.... and it ended up being more than just a little "tweak".
The simple fix:
You can improve the "scope" logic and cache the datablocks, stop replicating the cameras, and improve the logic so that fewer bytes are sent per object (i.e. do your best to only send updates for objects that have changed and only exactly what changed). The RTS Starter Kit has good examples on "network light" objects.
Using these techniques I was able to reduce Torque's bandwidth utilization by about 50% of stock Torque.
But, if that's not good enough, (as it wasn't for my needs)....
(continued)
#29
Scalability for games is no different than scalability for things like the middleware and exchanges used for the New York Stock Exchange, NASDAQ, etc. except with financial software you don't have the luxury of interpolating data, the volumes are higher, and you have more people "playing" simultaneously.
Writing software like this is my "day job" and I've written and maintained millions of lines of code related to solving this type of problem. After awhile you get a "feel" for the right answer.
Obviously there can be more than one "right" answer, but I'll outline my approach as one "right" answer among potentially many.
Completely replacing the datablock system is definitely part of the solution. You can do this if you approach it from the direction of replacing the ".mis" mission files and you get your "missions" or "zones" using some form of REST API + a data cache.
Depending on your game and your infrastructure, you can start out with something as simple as XML over HTTP (or HTTPS) for prototyping, and then progress on to compressed XML or EXI over HTTP(S). Using a standard protocol and standardized API makes it so that you can use standard software (Apache FTW!), and currently there's nothing that scales simpler and better than a bunch of web servers serving out static data. (and datablocks are nothing more than static data that might change periodically during "software / version" updates).
Taking advantage of the HTTP header that says "since date / time" allows you to provide incremental updates, too.
You have to be careful and not hang yourself with this as too many fine-grained requests or very large URL's could cause you to use more bandwidth and latency than what you'd think.
With the web server using a different pipe (and possibly some form of replication service such as Akamai or whatever) then you're not loading your zone servers or utilizing any of the zone server's bandwidth, and you can split the network and use less expensive higher latency pipes for the static data, and more expensive lower latency for the zone servers.
You can use Torque's HTTP object (or you might have to improve it to add HTTPS support) for this portion of the solution.
From there you have your choice of using the same type of setup for meshes, scripts, textures, etc, or you can have them pre-distributed with your installer, or you can use some form of P2P system to distribute it without using any of your bandwidth (don't forget to check the CRC's if you go this route), or you can use any combination of the above depending on your needs.
The key to this it that your zone server(s) are only used for actual dynamic data and object replication, not for acting as a data server for static data.
For smaller games such as an FPS game where you only have one server per game, using Torque as a data server was a viable solution, but for MMO games it's not the correct solution.
Everything else (login service, trade service, npc service, etc) should utilize some form of SOA where you split your services into individual bundles. If you write them in Java you can host them in something like Geronimo, IBM WebSphere, JBoss, etc. Or if you write them in C++ then you can use some other C++ application server (I'll get flamed if I mention the name, but I know of at least one that supports TCP and UDP networking with a customizable protocol stack and comes with a standard protocol stack for HTTP).
I have very successfully used this approach for both non-game and game related projects and it's actually more of a "defacto standard" architecture for most project requiring high scalability and reliability.
As for scalability, you can start out small and scale quickly and efficiently to extremely large numbers of web servers. It uses proven technology, and the bulk of the middleware is available as off-the-shelf software that you don't have to write. It's a perfect architecture for Indie (and commercial) game developers.
Although his article doesn't go into object replication improvements or datablock removal, Hyun Sung Chu of IBM was nice enough to publish an article on how you can use this type of architecture with Torque. If you're interested you can check it out here.
It's definitely a good start for allowing you to move your other services outside of the core game engine and into a full-sized application server.
But, that only solves a portion of the problem....
(continued)
02/02/2009 (10:16 am)
The "ultimate" solution:Scalability for games is no different than scalability for things like the middleware and exchanges used for the New York Stock Exchange, NASDAQ, etc. except with financial software you don't have the luxury of interpolating data, the volumes are higher, and you have more people "playing" simultaneously.
Writing software like this is my "day job" and I've written and maintained millions of lines of code related to solving this type of problem. After awhile you get a "feel" for the right answer.
Obviously there can be more than one "right" answer, but I'll outline my approach as one "right" answer among potentially many.
Completely replacing the datablock system is definitely part of the solution. You can do this if you approach it from the direction of replacing the ".mis" mission files and you get your "missions" or "zones" using some form of REST API + a data cache.
Depending on your game and your infrastructure, you can start out with something as simple as XML over HTTP (or HTTPS) for prototyping, and then progress on to compressed XML or EXI over HTTP(S). Using a standard protocol and standardized API makes it so that you can use standard software (Apache FTW!), and currently there's nothing that scales simpler and better than a bunch of web servers serving out static data. (and datablocks are nothing more than static data that might change periodically during "software / version" updates).
Taking advantage of the HTTP header that says "since date / time" allows you to provide incremental updates, too.
You have to be careful and not hang yourself with this as too many fine-grained requests or very large URL's could cause you to use more bandwidth and latency than what you'd think.
With the web server using a different pipe (and possibly some form of replication service such as Akamai or whatever) then you're not loading your zone servers or utilizing any of the zone server's bandwidth, and you can split the network and use less expensive higher latency pipes for the static data, and more expensive lower latency for the zone servers.
You can use Torque's HTTP object (or you might have to improve it to add HTTPS support) for this portion of the solution.
From there you have your choice of using the same type of setup for meshes, scripts, textures, etc, or you can have them pre-distributed with your installer, or you can use some form of P2P system to distribute it without using any of your bandwidth (don't forget to check the CRC's if you go this route), or you can use any combination of the above depending on your needs.
The key to this it that your zone server(s) are only used for actual dynamic data and object replication, not for acting as a data server for static data.
For smaller games such as an FPS game where you only have one server per game, using Torque as a data server was a viable solution, but for MMO games it's not the correct solution.
Everything else (login service, trade service, npc service, etc) should utilize some form of SOA where you split your services into individual bundles. If you write them in Java you can host them in something like Geronimo, IBM WebSphere, JBoss, etc. Or if you write them in C++ then you can use some other C++ application server (I'll get flamed if I mention the name, but I know of at least one that supports TCP and UDP networking with a customizable protocol stack and comes with a standard protocol stack for HTTP).
I have very successfully used this approach for both non-game and game related projects and it's actually more of a "defacto standard" architecture for most project requiring high scalability and reliability.
As for scalability, you can start out small and scale quickly and efficiently to extremely large numbers of web servers. It uses proven technology, and the bulk of the middleware is available as off-the-shelf software that you don't have to write. It's a perfect architecture for Indie (and commercial) game developers.
Although his article doesn't go into object replication improvements or datablock removal, Hyun Sung Chu of IBM was nice enough to publish an article on how you can use this type of architecture with Torque. If you're interested you can check it out here.
It's definitely a good start for allowing you to move your other services outside of the core game engine and into a full-sized application server.
But, that only solves a portion of the problem....
(continued)
#30
As for object replication improvements, for an MMO RPG with a quasi-reliable UDP layer for the "object replication", you can get away with publishing player "moves" instead of positions, and providing updates to positions once a second (should keep this as a "tuning parameter" though). Position updates should only be sent if the position changed since the last update.
Scoping can be done in such a way as to limit the number of dynamic objects a "player" can "see", this limits how many objects the zone server is having to replicate per player.
Using these techniques will allow you to reduce your bandwidth utilization to about 10% of the required "stock" Torque and improve your overall MMO game architecture.
02/02/2009 (10:19 am)
As for object replication improvements, for an MMO RPG with a quasi-reliable UDP layer for the "object replication", you can get away with publishing player "moves" instead of positions, and providing updates to positions once a second (should keep this as a "tuning parameter" though). Position updates should only be sent if the position changed since the last update.
Scoping can be done in such a way as to limit the number of dynamic objects a "player" can "see", this limits how many objects the zone server is having to replicate per player.
Using these techniques will allow you to reduce your bandwidth utilization to about 10% of the required "stock" Torque and improve your overall MMO game architecture.
#31
I also came across this thread here where Joel Raymond was using the same software used in some telephone switching applications (Erlang) to create their servers in a very scalable, fault-tolerant way. His findings were pretty surprising. It's very different from what most MMOs would be writing their servers in, but it's definitely something to look at.
02/02/2009 (2:03 pm)
@Tony: I had already come across the IBM whitepaper, and it has a really nice setup using Torque components. Your ideas on service architecture are spot on as well- online games are not so different from high-availability financial systems except in that the data is displayed as gameplay in MMOs, and graphs and other information in something like NASDAQ or Bloomberg. Even with the financial apps, they need to be able to communicate securely with the servers so that they can push their actions up to the network and make the necessary changes.I also came across this thread here where Joel Raymond was using the same software used in some telephone switching applications (Erlang) to create their servers in a very scalable, fault-tolerant way. His findings were pretty surprising. It's very different from what most MMOs would be writing their servers in, but it's definitely something to look at.
Torque 3D Owner Tony Richards
Correct me if I'm wrong, but doesn't WoW use it for distributing content updates, as do a whole lot of other commercial (non-game related) companies.
[edit - on topic only]