by date
Evolution of an Indie MMORPG
Evolution of an Indie MMORPG
| Name: | Prairie Games |  |
|---|---|---|
| Date Posted: | Jun 14, 2006 | |
| Rating: |  4.3 out of 5 | |
| Public: | YES | |
| Comments: | YES | |
| RSS Feed: |  or Subscribe with  . | |
| Profile Page: | View profile page for Prairie Games |
Blog post
Evolution of an Indie MMORPG
Minions of Mirth supports single player, multiplayer on custom community servers, and can be played as a MMORPG on dedicated servers hosted by Prairie Games, Inc. MoM has been live for 6 months and our community is steadily growing. I believe we can accelerate this growth, though we need to be prepared for it. I'll talk some about how we are preparing technically in this .plan.
This is a diagram of our original server layout. It's simplified in terms of (temporary) connections, the number of zone servers, etc. Though, it should give a reasonable overview.
Here's a short description of the various servers:
Master Server - This server handles registration of players and community operated servers. It handles authentication of "Free Editon" and "Premium Edition" accounts and routes players to worlds. Additionally, it holds a Character Server connection which is used in negotiating the Character Server to the World Server.
Character Server - This is a central repository for all characters played on Prairie Games, Inc servers. Players can use the same characters on any of our World Servers. This also allows "Free Edition" players to seamlessly upgrade to a "Premium Account" and not lose any progress.
World Server - This server runs all of the database and RPG logic for a world in a single process. Every zone available to players is connected to this process. Additionally, every player in the world is also connected and executes commands on this server.
Zone Server - a heavily modified TGE dedicated server. It communicates closely with the World Server for positional, animation, movement speed, spell effects, etc. It also holds connections to players for simulation in the 3d world. Players are routed by the World Server to the zone servers.
Client - The client is routed from the Master Server to a World Server. Once the player has chosen/created his character, a connection to a destination Zone Server is established and the player "zones into" the world. The player remains connected to the World Server via TCP and the Zone Server via UDP. The World Server connection is used for "RPG" type stuff and the Zone Server connection is used for simulation, selection of objects in the world, etc.
Summary - In the six months since launch, there have been various changes and optimizations to the MMORPG component of the game. It's currently performing well, though there is definitely room for improvement. In order to serve a substantially larger player base, the performance needs to be scaled. The obvious bottleneck being the World Server, which again is served in a single process for all connected zones and players.
Ok, so the World Server bottleneck had to be solved. Here's the new implementation, which we'll be rolling out within the next couple weeks. Again, it's a bit simplified:
Here are the changes of note:
Zone Cluster Server - The single world process of the old system has been broken up into distributed Zone Cluster processes. This allows multiple CPU's (and machines) to be utilized and concurrent processing of players, zones, RPG, and database logic. A cluster can be configured to serve an arbitrary number of zones and these can be intelligently chosen. For instance, coupling a busy zone with a zone that gets little traffic is a good strategy. I also added additional checks based on individual player CPU time. This way someone who is macroing expensive operations can't tie up the server, etc.
World Daemon - This is a single point of connection between a World and the Master and Character servers. It coordinates the Zone Clusters that make up a discrete world and routes players between them. It contains logic for chat, and other game elements that must span clusters. The World Daemon is also used as a data replication service for backing up character data. Furthermore, it can be configured to reboot itself at off peak times to help keep the servers running smoothly.
Client - The client now jumps between Zone Cluster servers. This posed a number of issues with features like player alliances. Though, it was mostly smooth to implement.
Summary - We'll be able to support many more players with our existing servers by distributing the World Server process across multiple processes, CPU's, and even machines. The Zone Cluster architecture is scaleable and critical for adding new zones to the game's existing content. It was relatively painless to implement these changes to the backend. It will be extremely interesting to see this go live.
-Josh Ritter
Prairie Games, Inc
http://www.prairiegames.com
Minions of Mirth supports single player, multiplayer on custom community servers, and can be played as a MMORPG on dedicated servers hosted by Prairie Games, Inc. MoM has been live for 6 months and our community is steadily growing. I believe we can accelerate this growth, though we need to be prepared for it. I'll talk some about how we are preparing technically in this .plan.
This is a diagram of our original server layout. It's simplified in terms of (temporary) connections, the number of zone servers, etc. Though, it should give a reasonable overview.
Here's a short description of the various servers:
Master Server - This server handles registration of players and community operated servers. It handles authentication of "Free Editon" and "Premium Edition" accounts and routes players to worlds. Additionally, it holds a Character Server connection which is used in negotiating the Character Server to the World Server.
Character Server - This is a central repository for all characters played on Prairie Games, Inc servers. Players can use the same characters on any of our World Servers. This also allows "Free Edition" players to seamlessly upgrade to a "Premium Account" and not lose any progress.
World Server - This server runs all of the database and RPG logic for a world in a single process. Every zone available to players is connected to this process. Additionally, every player in the world is also connected and executes commands on this server.
Zone Server - a heavily modified TGE dedicated server. It communicates closely with the World Server for positional, animation, movement speed, spell effects, etc. It also holds connections to players for simulation in the 3d world. Players are routed by the World Server to the zone servers.
Client - The client is routed from the Master Server to a World Server. Once the player has chosen/created his character, a connection to a destination Zone Server is established and the player "zones into" the world. The player remains connected to the World Server via TCP and the Zone Server via UDP. The World Server connection is used for "RPG" type stuff and the Zone Server connection is used for simulation, selection of objects in the world, etc.
Summary - In the six months since launch, there have been various changes and optimizations to the MMORPG component of the game. It's currently performing well, though there is definitely room for improvement. In order to serve a substantially larger player base, the performance needs to be scaled. The obvious bottleneck being the World Server, which again is served in a single process for all connected zones and players.
Ok, so the World Server bottleneck had to be solved. Here's the new implementation, which we'll be rolling out within the next couple weeks. Again, it's a bit simplified:
Here are the changes of note:
Zone Cluster Server - The single world process of the old system has been broken up into distributed Zone Cluster processes. This allows multiple CPU's (and machines) to be utilized and concurrent processing of players, zones, RPG, and database logic. A cluster can be configured to serve an arbitrary number of zones and these can be intelligently chosen. For instance, coupling a busy zone with a zone that gets little traffic is a good strategy. I also added additional checks based on individual player CPU time. This way someone who is macroing expensive operations can't tie up the server, etc.
World Daemon - This is a single point of connection between a World and the Master and Character servers. It coordinates the Zone Clusters that make up a discrete world and routes players between them. It contains logic for chat, and other game elements that must span clusters. The World Daemon is also used as a data replication service for backing up character data. Furthermore, it can be configured to reboot itself at off peak times to help keep the servers running smoothly.
Client - The client now jumps between Zone Cluster servers. This posed a number of issues with features like player alliances. Though, it was mostly smooth to implement.
Summary - We'll be able to support many more players with our existing servers by distributing the World Server process across multiple processes, CPU's, and even machines. The Zone Cluster architecture is scaleable and critical for adding new zones to the game's existing content. It was relatively painless to implement these changes to the backend. It will be extremely interesting to see this go live.
-Josh Ritter
Prairie Games, Inc
http://www.prairiegames.com
Recent Blog Posts
| List: | 03/29/08 - TGEA 1.7 Build System and Embedded Python 03/14/08 - MegaTerrains - TGEA Update 01/18/08 - Minions of Mirth: Undead Wars Expansion 01/04/08 - Physics Overhaul - Video 12/26/07 - Web Integration - Video 12/21/07 - New MMO Client - Trees - Day/Night Video 12/18/07 - Minions of Mirth - 1.26 - Holiday Edition! 11/28/07 - TGB/TGEA integration first pass |
|---|
Submit your own resources!| Jonathon Stevens (Jun 14, 2006 at 17:31 GMT) |
| Frank Geist (Jun 14, 2006 at 17:40 GMT) Resource Rating: 5 |
Keep up the great work, please!
| Andy Schatz (Jun 14, 2006 at 17:47 GMT) Resource Rating: 5 |
| Prairie Games (Jun 14, 2006 at 18:44 GMT) Resource Rating: 5 |
We average around 60 players online off-peak with a peak of around 100. We're probably at about 50% of our current ceiling so let's say that the current architecture supports 200 simultaneous players. I predict that we'll be able to (easily) handle 2-3x that or around 400-600 concurrent logins... from here, there are some other optimizations which could bring that up further, or 400 per physical server. That's pretty good I think... it'll be interesting to examine the load on the live servers. 400 MoM players in a single world will also affect the gameplay and these are only technical considerations. I am not a fan of instanced zones, but the new architecture would readily support this if needed.
-JR
| Hans (Jun 14, 2006 at 19:22 GMT) |
| Prairie Games (Jun 14, 2006 at 19:38 GMT) Resource Rating: 5 |
I haven't actually seen the PC Gamer review, but any score is fine as long as we get in the magazine!
| Jonathon Stevens (Jun 14, 2006 at 19:54 GMT) |
| Ian \"Xest\" Winter (Jun 14, 2006 at 20:21 GMT) |
It's interesting to see your methods of spreading the load Josh, distributing Torque is something I've been considering having a go at myself but time constraints have held me back so far so these blogs are a fantastic insight into your method of doing it ;)
| Andy Schatz (Jun 14, 2006 at 23:06 GMT) Resource Rating: 5 |
| Mike Stoddart (Jun 14, 2006 at 23:15 GMT) Resource Rating: 5 |
| Prairie Games (Jun 14, 2006 at 23:35 GMT) Resource Rating: 5 |
@Mike: Thanks for your support :)
@Ian: The LBA games are a couple of my favorite.
@Andy: The city zones often have 20-40 people in them. The metrics I give above are for 2 dual Xeon servers working in cooperation. The new architecture will allow us to have between 200-300 on a single physical server. There are further optimizations which could bring this closer to 400. We're looking at being able to support 600-800 simultaneous players with our 2 main servers.
The game also has multiple chat channels that span worlds and players can travel freely between them with the same characters. So, we can always add servers to support more players.
-JR
Edited on Jun 14, 2006 23:38 GMT
| Unsung Zero (Jun 15, 2006 at 20:16 GMT) |
While I'm looking forward to the upgrade, I'm worried on how the global channel will work out with 400+ players sharing it. It's kind of busy as it is at times.
| Oliver Rendelmann - DerR (Jun 20, 2006 at 15:00 GMT) Resource Rating: 5 |
By the way... I would of bought MoM already but I live in germany and the lag is too big to truely play it nicely. :(
| Matthew Overlund (Jul 13, 2006 at 18:05 GMT) |
You may have posted this data before, but I missed it if so. What kind of network pipe are you running on the data center?
| Sparkling (Jan 25, 2007 at 19:28 GMT) Resource Rating: 4 |
Blessings to you!
-Sparkling
http://www.visionsgame.com
Edited on Jan 25, 2007 19:31 GMT
You must be a member and be logged in to either append comments or rate this resource.
