Evolution of an Indie MMORPG - Part 2
by Prairie Games · 10/24/2006 (4:11 pm) · 7 comments
Evolution of an Indie MMORPG - Part 2
I've been working on the next iteration of the MoM server architecture. The primary goal of the last version was in distributing the world server across multiple processors. The important feature being added this time is the ability to serve a world across multiple physical machines. This further distributes the server load and allows us to increase the world's size in terms of zones. We're also working on increasing the server's performance including moving a bunch of logic to the client (with verification on the server).
The diagram below shows the new server layout. The large white boxes represent physical machines that contain a number of logical servers. What's being illustrated are two independent worlds each being spread across 3 physical machines. As the world's are connected to the same character server, players can actually choose to play on either of them. For instance, premium players can log onto the free world with their friends who haven't purchased the game.
I'll describe the various servers very briefly:
Master Server- The master server handles account registration and authenticating players with the patch server and world servers (both player operated and official).
Character Server- We store the characters for the official worlds on this server. We also store guild information and other "global player" data here. It would be possible to open up a number of Character Servers for additional shards perhaps based on things like the language being used in chat (spanish, german, french, etc)
GM Server- GM's log into this server to help administer the game. We have a "GM Tool" application that allows them to inspect characters, kick/mute/ban abusive players, etc. We plan for this tool to also help run GM Events and other fun activities.
World Daemon- This server is used to coordinate and communicate between a number of zone clusters. It also has basic process control for rebooting and shutting down a world. Worlds automatically reboot themselves daily at 3am PST. This process takes less than 5 minutes and keeps everything running smoothly.
Zone Cluster- All of the RPG logic is handled in the zone cluster processes. This means that the simulation is never stalled for a database access, or other time consuming event. The server is kind of a "puppet master" for a number of connected zone (simulation) servers. Players are dynamically passed between zone clusters and zone servers on the fly as they move between zones in the world.
Zone Server- This is a modified Torque Game Engine dedicated server. I've added a number of features and optimizations important to MMORPG styled games. The zone server and zone cluster work closely over the wire... this includes passing positional data, zone weather, spell effects, etc
World Imp- A server that allows control over a number of zone clusters which are remotely controlled by a world daemon. This and modifications to the zone cluster code itself allow a single world to be served across a number of physical machines (and processors within the same machine). We can now scale the world up to a very large number of zones and distribute these zones based on the amount of traffic they receive. We can also efficiently setup zone instancing if it was ever desired...
Minions of Mirth now scales from single player with an embedded world server "in the client" to a MMORPG with world servers that are distributed across multiple physical machines and effectively use multiple processors. The latest changes are quite significant in terms of performance. We're also aggressively optimizing the server code. It'll feel good to get this latest version of the technology live. It will be in testing very soon. :)
-Josh Ritter
President
Prairie Games, Inc
I've been working on the next iteration of the MoM server architecture. The primary goal of the last version was in distributing the world server across multiple processors. The important feature being added this time is the ability to serve a world across multiple physical machines. This further distributes the server load and allows us to increase the world's size in terms of zones. We're also working on increasing the server's performance including moving a bunch of logic to the client (with verification on the server).
The diagram below shows the new server layout. The large white boxes represent physical machines that contain a number of logical servers. What's being illustrated are two independent worlds each being spread across 3 physical machines. As the world's are connected to the same character server, players can actually choose to play on either of them. For instance, premium players can log onto the free world with their friends who haven't purchased the game.
I'll describe the various servers very briefly:
Master Server- The master server handles account registration and authenticating players with the patch server and world servers (both player operated and official).
Character Server- We store the characters for the official worlds on this server. We also store guild information and other "global player" data here. It would be possible to open up a number of Character Servers for additional shards perhaps based on things like the language being used in chat (spanish, german, french, etc)
GM Server- GM's log into this server to help administer the game. We have a "GM Tool" application that allows them to inspect characters, kick/mute/ban abusive players, etc. We plan for this tool to also help run GM Events and other fun activities.
World Daemon- This server is used to coordinate and communicate between a number of zone clusters. It also has basic process control for rebooting and shutting down a world. Worlds automatically reboot themselves daily at 3am PST. This process takes less than 5 minutes and keeps everything running smoothly.
Zone Cluster- All of the RPG logic is handled in the zone cluster processes. This means that the simulation is never stalled for a database access, or other time consuming event. The server is kind of a "puppet master" for a number of connected zone (simulation) servers. Players are dynamically passed between zone clusters and zone servers on the fly as they move between zones in the world.
Zone Server- This is a modified Torque Game Engine dedicated server. I've added a number of features and optimizations important to MMORPG styled games. The zone server and zone cluster work closely over the wire... this includes passing positional data, zone weather, spell effects, etc
World Imp- A server that allows control over a number of zone clusters which are remotely controlled by a world daemon. This and modifications to the zone cluster code itself allow a single world to be served across a number of physical machines (and processors within the same machine). We can now scale the world up to a very large number of zones and distribute these zones based on the amount of traffic they receive. We can also efficiently setup zone instancing if it was ever desired...
Minions of Mirth now scales from single player with an embedded world server "in the client" to a MMORPG with world servers that are distributed across multiple physical machines and effectively use multiple processors. The latest changes are quite significant in terms of performance. We're also aggressively optimizing the server code. It'll feel good to get this latest version of the technology live. It will be in testing very soon. :)
-Josh Ritter
President
Prairie Games, Inc
#2
Now if I could just figure out how to kill one of your Sickly Skeletons without losing half my health. :-)
10/24/2006 (4:41 pm)
Wow. Very impressive!Now if I could just figure out how to kill one of your Sickly Skeletons without losing half my health. :-)
#3
10/25/2006 (1:41 am)
Allways a pleasure reading your postings, keep it up Josh ;) 
#4
10/25/2006 (9:20 am)
Good work Josh, it's nice to see MoM progress with demand.
#5
10/25/2006 (12:02 pm)
Lots of good info in here for anyone looking at either researching, or implementing, back end server architecture. I came up with pretty much the exact architecture myself for various support requests from commercial support clients, and he's nailed it all in one post--great job!
#6
Myself, and the guy next to me, both keen bass players, looked at each other and said "Why do we bother?", so impressed were we. I have not played since.
I kinda get the same feeling when you post Josh, but I'm trying to look at it as inspiration for my own efforts :-)
Great work, nice concise explanation.
10/25/2006 (5:59 pm)
I remember many years ago going to a Satriani concert with some mates. Wasnt impressed that much untill the bassist, Stu Ham I think, played the theme to star trek on the bass, solo.Myself, and the guy next to me, both keen bass players, looked at each other and said "Why do we bother?", so impressed were we. I have not played since.
I kinda get the same feeling when you post Josh, but I'm trying to look at it as inspiration for my own efforts :-)
Great work, nice concise explanation.
#7
Now have you thought of moving some of the patching and pre-processing out to the "IMP" servers to reduce the load on the master server?
I've thought about this for a future project of mine when I get to the MMO stage's of programming ( a few years away still :( ).
As I thought if as in your description if the IMP server act'd like an vitual Master server handling client patching, pre-processing (of things like character management [equipment/weapons, spells, etc...], and other related things, could free the master to manage things like character movement between regions, combat, item trading, core chat/comm's systems, etc..
In an sence the "IMP" would be the local master server for it's zone and only need to pass onto the Master server where the character is and it's condition (what's it has armed and doing, and possibly it's current health) and any extra data that maybe required depending on the character's current action (trading with NPC or other player, then updated equipment data sent to master server database), and any chat/comm's at an higher compression than to/from client to "IMP".
Have you any thoughts on this?
01/16/2007 (1:25 am)
Josh, just an question.Now have you thought of moving some of the patching and pre-processing out to the "IMP" servers to reduce the load on the master server?
I've thought about this for a future project of mine when I get to the MMO stage's of programming ( a few years away still :( ).
As I thought if as in your description if the IMP server act'd like an vitual Master server handling client patching, pre-processing (of things like character management [equipment/weapons, spells, etc...], and other related things, could free the master to manage things like character movement between regions, combat, item trading, core chat/comm's systems, etc..
In an sence the "IMP" would be the local master server for it's zone and only need to pass onto the Master server where the character is and it's condition (what's it has armed and doing, and possibly it's current health) and any extra data that maybe required depending on the character's current action (trading with NPC or other player, then updated equipment data sent to master server database), and any chat/comm's at an higher compression than to/from client to "IMP".
Have you any thoughts on this?
Torque Owner Stephan - viKKing - Bondier
Josh you are really killing me with each of your blog updates. Please stop posting!
Just kidding. Your posts are really a deep breathless dive into MMORPG technology.
Any significant update on the number of players?
STef