Evolution of an Indie MMORPG
by Prairie Games · 06/14/2006 (9:41 am) · 15 comments
Evolution of an Indie MMORPG
Minions of Mirth supports single player, multiplayer on custom community servers, and can be played as a MMORPG on dedicated servers hosted by Prairie Games, Inc. MoM has been live for 6 months and our community is steadily growing. I believe we can accelerate this growth, though we need to be prepared for it. I'll talk some about how we are preparing technically in this .plan.
This is a diagram of our original server layout. It's simplified in terms of (temporary) connections, the number of zone servers, etc. Though, it should give a reasonable overview.
Here's a short description of the various servers:
Master Server - This server handles registration of players and community operated servers. It handles authentication of "Free Editon" and "Premium Edition" accounts and routes players to worlds. Additionally, it holds a Character Server connection which is used in negotiating the Character Server to the World Server.
Character Server - This is a central repository for all characters played on Prairie Games, Inc servers. Players can use the same characters on any of our World Servers. This also allows "Free Edition" players to seamlessly upgrade to a "Premium Account" and not lose any progress.
World Server - This server runs all of the database and RPG logic for a world in a single process. Every zone available to players is connected to this process. Additionally, every player in the world is also connected and executes commands on this server.
Zone Server - a heavily modified TGE dedicated server. It communicates closely with the World Server for positional, animation, movement speed, spell effects, etc. It also holds connections to players for simulation in the 3d world. Players are routed by the World Server to the zone servers.
Client - The client is routed from the Master Server to a World Server. Once the player has chosen/created his character, a connection to a destination Zone Server is established and the player "zones into" the world. The player remains connected to the World Server via TCP and the Zone Server via UDP. The World Server connection is used for "RPG" type stuff and the Zone Server connection is used for simulation, selection of objects in the world, etc.
Summary - In the six months since launch, there have been various changes and optimizations to the MMORPG component of the game. It's currently performing well, though there is definitely room for improvement. In order to serve a substantially larger player base, the performance needs to be scaled. The obvious bottleneck being the World Server, which again is served in a single process for all connected zones and players.
Ok, so the World Server bottleneck had to be solved. Here's the new implementation, which we'll be rolling out within the next couple weeks. Again, it's a bit simplified:
Here are the changes of note:
Zone Cluster Server - The single world process of the old system has been broken up into distributed Zone Cluster processes. This allows multiple CPU's (and machines) to be utilized and concurrent processing of players, zones, RPG, and database logic. A cluster can be configured to serve an arbitrary number of zones and these can be intelligently chosen. For instance, coupling a busy zone with a zone that gets little traffic is a good strategy. I also added additional checks based on individual player CPU time. This way someone who is macroing expensive operations can't tie up the server, etc.
World Daemon - This is a single point of connection between a World and the Master and Character servers. It coordinates the Zone Clusters that make up a discrete world and routes players between them. It contains logic for chat, and other game elements that must span clusters. The World Daemon is also used as a data replication service for backing up character data. Furthermore, it can be configured to reboot itself at off peak times to help keep the servers running smoothly.
Client - The client now jumps between Zone Cluster servers. This posed a number of issues with features like player alliances. Though, it was mostly smooth to implement.
Summary - We'll be able to support many more players with our existing servers by distributing the World Server process across multiple processes, CPU's, and even machines. The Zone Cluster architecture is scaleable and critical for adding new zones to the game's existing content. It was relatively painless to implement these changes to the backend. It will be extremely interesting to see this go live.
-Josh Ritter
Prairie Games, Inc
http://www.prairiegames.com
Minions of Mirth supports single player, multiplayer on custom community servers, and can be played as a MMORPG on dedicated servers hosted by Prairie Games, Inc. MoM has been live for 6 months and our community is steadily growing. I believe we can accelerate this growth, though we need to be prepared for it. I'll talk some about how we are preparing technically in this .plan.
This is a diagram of our original server layout. It's simplified in terms of (temporary) connections, the number of zone servers, etc. Though, it should give a reasonable overview.
Here's a short description of the various servers:
Master Server - This server handles registration of players and community operated servers. It handles authentication of "Free Editon" and "Premium Edition" accounts and routes players to worlds. Additionally, it holds a Character Server connection which is used in negotiating the Character Server to the World Server.
Character Server - This is a central repository for all characters played on Prairie Games, Inc servers. Players can use the same characters on any of our World Servers. This also allows "Free Edition" players to seamlessly upgrade to a "Premium Account" and not lose any progress.
World Server - This server runs all of the database and RPG logic for a world in a single process. Every zone available to players is connected to this process. Additionally, every player in the world is also connected and executes commands on this server.
Zone Server - a heavily modified TGE dedicated server. It communicates closely with the World Server for positional, animation, movement speed, spell effects, etc. It also holds connections to players for simulation in the 3d world. Players are routed by the World Server to the zone servers.
Client - The client is routed from the Master Server to a World Server. Once the player has chosen/created his character, a connection to a destination Zone Server is established and the player "zones into" the world. The player remains connected to the World Server via TCP and the Zone Server via UDP. The World Server connection is used for "RPG" type stuff and the Zone Server connection is used for simulation, selection of objects in the world, etc.
Summary - In the six months since launch, there have been various changes and optimizations to the MMORPG component of the game. It's currently performing well, though there is definitely room for improvement. In order to serve a substantially larger player base, the performance needs to be scaled. The obvious bottleneck being the World Server, which again is served in a single process for all connected zones and players.
Ok, so the World Server bottleneck had to be solved. Here's the new implementation, which we'll be rolling out within the next couple weeks. Again, it's a bit simplified:
Here are the changes of note:
Zone Cluster Server - The single world process of the old system has been broken up into distributed Zone Cluster processes. This allows multiple CPU's (and machines) to be utilized and concurrent processing of players, zones, RPG, and database logic. A cluster can be configured to serve an arbitrary number of zones and these can be intelligently chosen. For instance, coupling a busy zone with a zone that gets little traffic is a good strategy. I also added additional checks based on individual player CPU time. This way someone who is macroing expensive operations can't tie up the server, etc.
World Daemon - This is a single point of connection between a World and the Master and Character servers. It coordinates the Zone Clusters that make up a discrete world and routes players between them. It contains logic for chat, and other game elements that must span clusters. The World Daemon is also used as a data replication service for backing up character data. Furthermore, it can be configured to reboot itself at off peak times to help keep the servers running smoothly.
Client - The client now jumps between Zone Cluster servers. This posed a number of issues with features like player alliances. Though, it was mostly smooth to implement.
Summary - We'll be able to support many more players with our existing servers by distributing the World Server process across multiple processes, CPU's, and even machines. The Zone Cluster architecture is scaleable and critical for adding new zones to the game's existing content. It was relatively painless to implement these changes to the backend. It will be extremely interesting to see this go live.
-Josh Ritter
Prairie Games, Inc
http://www.prairiegames.com
#2
Keep up the great work, please!
06/14/2006 (10:40 am)
Wow! I've been playing MoM for a couple months now. Your work is nothing short of mindblowing. It's very inspiring to a MMORPG maker wannabe like me :)Keep up the great work, please!
#3
06/14/2006 (10:47 am)
WOuld you mind sharing player numbers? How many do you support on your current architecture and how many do you plan to be able to support on the new architecture? Of course only if you're comfortable releasing the data...
#4
We average around 60 players online off-peak with a peak of around 100. We're probably at about 50% of our current ceiling so let's say that the current architecture supports 200 simultaneous players. I predict that we'll be able to (easily) handle 2-3x that or around 400-600 concurrent logins... from here, there are some other optimizations which could bring that up further, or 400 per physical server. That's pretty good I think... it'll be interesting to examine the load on the live servers. 400 MoM players in a single world will also affect the gameplay and these are only technical considerations. I am not a fan of instanced zones, but the new architecture would readily support this if needed.
-JR
06/14/2006 (11:44 am)
@Andy: Our servers are dual Xeons and one of the processors is barely utilized. There's also the issue with a single process getting backlogged if there is any amount of CPU spike. This is actually a major consideration. By distributing the load across multiple processes, this will be less frequent and the process will be able to catch back up faster. Right now, the simulation never lags (14 zone processes are distributing the work). The rpg side of things, on the world server, can spike here and there.We average around 60 players online off-peak with a peak of around 100. We're probably at about 50% of our current ceiling so let's say that the current architecture supports 200 simultaneous players. I predict that we'll be able to (easily) handle 2-3x that or around 400-600 concurrent logins... from here, there are some other optimizations which could bring that up further, or 400 per physical server. That's pretty good I think... it'll be interesting to examine the load on the live servers. 400 MoM players in a single world will also affect the gameplay and these are only technical considerations. I am not a fan of instanced zones, but the new architecture would readily support this if needed.
-JR
#5
06/14/2006 (12:22 pm)
I saw yesterday a review in PC Gamer Mag of MOM they gave it a 50% the review was very short.
#6
I haven't actually seen the PC Gamer review, but any score is fine as long as we get in the magazine!
06/14/2006 (12:38 pm)
Yeah, I was a bit surprised that they reviewed it... pleased though :)I haven't actually seen the PC Gamer review, but any score is fine as long as we get in the magazine!
#7
06/14/2006 (12:54 pm)
Josh, have you undergone any major marketing pushes at all? Are you planning another post mortum article stating more of the business side of MOM (huh, I just noticed that MOM is WOW - world of warcraft - upside down) ;)
#8
It's interesting to see your methods of spreading the load Josh, distributing Torque is something I've been considering having a go at myself but time constraints have held me back so far so these blogs are a fantastic insight into your method of doing it ;)
06/14/2006 (1:21 pm)
50% seems a bit harsh, but then I guess PC Magazines don't take into account that you didn't have a $50million budget when you made the game which is a little unfair, perhaps it's time for PC magazines to evolve too and have an indie section which realises the fact that indies are never going to be producing games equivalent to those with massive budgets? As you say though Josh, any review is a good review. I remember many many years ago seeing that old game, "Little Big Adventure" and "Little Big Adventure 2" reviewed in a magazine too, it only just broke 50% in most magazines if I recall right yet it was one of the best games of it's time, the graphics were superb for it's era, and still look good now IMHO (http://el-muerte.student.utwente.nl/lba/image_galleries/lba2/screenshots), the gameplay was fantastic, the idea and storyline were brilliant so I've never really thought magazine reviews were worth the paper they were written on anyhow ;)It's interesting to see your methods of spreading the load Josh, distributing Torque is something I've been considering having a go at myself but time constraints have held me back so far so these blogs are a fantastic insight into your method of doing it ;)
#9
06/14/2006 (4:06 pm)
Interesting. But if I understand your architecture correctly, do these people have to be distributed across zone for this to work? How many can it support in a zone, and how many concurrent in a world (distributed across the zones)? This is interesting stuff to me. I've done a bunch of multiplayer games, but never an MMO.
#10
06/14/2006 (4:15 pm)
I feel bad that I don't have time to play MoM, but I bought the game so I'm supporting it in one way!
#11
@Mike: Thanks for your support :)
@Ian: The LBA games are a couple of my favorite.
@Andy: The city zones often have 20-40 people in them. The metrics I give above are for 2 dual Xeon servers working in cooperation. The new architecture will allow us to have between 200-300 on a single physical server. There are further optimizations which could bring this closer to 400. We're looking at being able to support 600-800 simultaneous players with our 2 main servers.
The game also has multiple chat channels that span worlds and players can travel freely between them with the same characters. So, we can always add servers to support more players.
-JR
06/14/2006 (4:35 pm)
@dishmal: There have been no marketing pushes of any strength@Mike: Thanks for your support :)
@Ian: The LBA games are a couple of my favorite.
@Andy: The city zones often have 20-40 people in them. The metrics I give above are for 2 dual Xeon servers working in cooperation. The new architecture will allow us to have between 200-300 on a single physical server. There are further optimizations which could bring this closer to 400. We're looking at being able to support 600-800 simultaneous players with our 2 main servers.
The game also has multiple chat channels that span worlds and players can travel freely between them with the same characters. So, we can always add servers to support more players.
-JR
#12
While I'm looking forward to the upgrade, I'm worried on how the global channel will work out with 400+ players sharing it. It's kind of busy as it is at times.
06/15/2006 (1:16 pm)
The new setup looks good :] It should be pretty scalable like that.While I'm looking forward to the upgrade, I'm worried on how the global channel will work out with 400+ players sharing it. It's kind of busy as it is at times.
#13
By the way... I would of bought MoM already but I live in germany and the lag is too big to truely play it nicely. :(
06/20/2006 (8:00 am)
Looks very interesting! I'm also a MMORPG wannabe.. not because I want to make big money, but because I think it's extremely interesting on the technical side! Are you going to post more inside info on that? And would you mind if I contact you personally for questions?By the way... I would of bought MoM already but I live in germany and the lag is too big to truely play it nicely. :(
#14
You may have posted this data before, but I missed it if so. What kind of network pipe are you running on the data center?
07/13/2006 (11:05 am)
Josh,You may have posted this data before, but I missed it if so. What kind of network pipe are you running on the data center?
#15
Blessings to you!
-Sparkling
http://www.visionsgame.com
01/25/2007 (11:28 am)
I'm (trying to) make a MMORPG also, not nearly this far along in the process yet though. I salute your accomplishment! And we are doing it with TSE instead of TGE so it's a little different. But when I look at your server structure (knowing what issues we have had already with lag on our 1 little server) all I can think of is BANDWIDTH PLEASE!!! I'm curious, do you have a dedicated FIOS line to each one of those server boxes? How do you manage that? Our experience so far has been that the game can handle more than the bandwidth can. And bandwidth doesn't come cheap.Blessings to you!
-Sparkling
http://www.visionsgame.com
Torque 3D Owner Jonathon Stevens