V12 Technology and Massively Multiplayer Online Game

#21 Ok this is the third time I am going to try and post this, I tell you there is a conspiracy afoot :)

First, I am not a programmer so I am not sure how close the network codes are, I do know that my programmers are compiling code on both system, and I beleive what they need to is changes some of the headers for the C++ files. Other then that I am not sure from a C programmers perspective, maybe I can get one of them to post and let you know.

As in the Cluster configurations I am sure that they can handle to load, but we will not know until we try in a real application. Maybe we should write Microsoft and ask them the spec of their openBSD systems handle the load for hotmail, since they have been using is for several years now.

Here is the configurations that we plan on testing on a small scale once we get the code stable.

5 ultra sparc's with 400 mhz's processors and 1 gig of ram, now will this server fit hundreds of thousands of users, no but its where we are going to start benchmarking.

The operating system we are going to use is Redhat Linux optimized for the ultrasparc arhitecture. I will be more then happy to share our results with the community.

#22 Just thought I would throw this out to you as more of an informational link (not taking sides as to best NOS or CPU, but I have my own opinion):

http://clusters.top500.org/

http://top500.org/

These are mostly for research but alot of basic info like NOS/support software and cluster speed (but most only listed as FLOPS), also alot of them have webpage for more information.

#23 Guys, I wasnt saying linux was or wasnt suitable for a MMOG, I was just trying to say that it takes more than putting a server
executable onto a server and running it.

For truly large scale games, you are AT LEAST looking at clustering, distributed database of some sort, probably hardware
load balancing and security issues, this is BEFORE you start working on the "game" specific issues.

We are not talking a small change from the tribes model.. we are talking a whole new world.

I just get a bit frustrated by people who think its possible to take a 16-100 player model, and turn it into a 1000-10,000 player model
without SERIOUS work.

I'll leave the platform decisions to those more informed than myself, from what I'm aware, the only single factor for the
commercial MMOG out there is that they seem to have all used oracle for thier database.

I dont know if thats true for Asherons call though, they seemed to have used NT (if you believe thier peice on gamasutra?).

Take my advice, steer clear of NT :)

Phil

#24 Well I agree with Phil on the fact that a program written to run 10-100 clients is more than likely not going to be able to support 1000-10000 clients. But the reality is that I think what NOS you run is less important than the quality of the code written to run on the server(s).

If you write a single-threaded application and run it on the largest Unix/linux/W2K cluster available today you will not be able to support near the number of clients in comparison to a well planned multi-threaded application running on a efficient cluster of servers.

In regard's to X NOS cant handle the load that Y NOS can, each operating system has its strong points and each project should weigh the pro's and con's of each for their preferred solution. But IMO the best solution may infact be a combination of NOS's so you can draw on the benefit's of each Dplay(W2K) interface for lobby/player management with linux/Unix back end for DB and number crunching.

Anyway, I don't have a server model complete, but I am looking forward to doing more research and refining my opinion on the best solution for a MMORPG server model.

#25 Hay regarding databases,

Forget Oracle, take a look at MySQL.

www.mysql.com

The Pros:

Open Source / GNU
Free
Linux, Windows, Sun, etc.. compatible
One of the fastest databases in the world
Nice C/C++ API

Cons:

No native atomic operations yet [you do things like referential integrity yourself through programming].

Only just starting to support row level locking, hence it's concurrency might not be as good as some databases when exceeding a couple hundred simultaneous connections but a beta version is available that may help the concurrency issue.

Some of the biggest websites in the world supposedly use MySQL, also take a look at Postgress SQL for linux as well.. You do not have to spend tens of thousands on Oracle or IBM based solutions if you dont want to...

Mike

#26 Only because I have been knee deep at Exodus recently, I bring you my pseudo-random collection of thoughts on the subject. And, yes, I build datacenters as a day job ;)

As far as Databases, Oracle or SQL 2000. MySQL has a laughable security model and ALL of the referential integrity needs to be handled in code. AND, it simply doesn't ahve the scalibilty and performance statistics to be a proven contender. It might get there, and there MIGHT be flavors that can handle the load, but what you really need is a performant, scalable and reliable database platform.

Not sure what postgres offers in the way of clustering, but, Oracle and SQL both offer out of the box clustering solutions that work for fail over and (to some extent) load balancing between the cluster nodes. Early incarnations were only good at clustering for hot failover, but things like Oracle Parallel Server and MS Clustering Service have finally matured into products that can be used for both failover and load balancing AT THE DATABASE LEVEL.

This is still not clustering at the application level or load balancing at the application level, this is simply the database. Very few things beat the Solaris/Oracle combo (unless you believe the MS hype, which, if reports I get from the field are accurate, it may be spot on).

Someone above said "software is cheap, hardware is expensive" I'd partially aggree with that. Hardware for the levels of scalability, reliability and performance at the levels desired, all cost about the same. Think about you good high end Linux server box. IBM is gonna charge you the same amount pretty much if you put Solaris Intel/Oracle, Linux/Postgres, NT/SQL Svr or AIX and DB2 on it. Memory costs the same, procs cost the same, redundant power supplies cost the same. When you get into highly available, scalable systems, even the Sun vs IBM vs Intel difference diminishes to the point of "who cares, it's costing me a fortune anyway."

Now, clustering and load balancing at the application level is another thing entirely. Most Beowulf applications need substantial porting to be able to run on a Cluster. Messaging between the nodes is complicated, and, if you don't have the first principles down on TRUE clusters, it's easy to muck up (be a cool experiment though). The Beowulf cluster truly acts as one large machine, sharing processing of tasks out over the multiple nodes in the cluster and presenting a result to the client. Great for complicated on the fly math, weather modelling, and modelling of other dynamic systems or brute force number crunching (the likes of Seti@Home). I would be PSYCHED to see a commercial entertainment application for this (CAVE comes to mind) but, I am not sure of it's suitability here.

That leaves clustering for fail over (much like the db servers above) and Load Balancing of client connects. Take the web for an example. All of the datacenters I have built have utilized dedicated hardware load balancers, either F5 BigIP/3DNS or Ciscos Local Director.

Raw TCP/UDP connects are redirected by rules in the device to the application/web server of choice for that moment. Indivisual client connects are either dymanic (meaning it doesn't matter which server in teh farm the visit) or persistent (meaning, once you connect to server A you stay connected to server A for the duration of the session). Sometimes state is managed at the database level, so you don't care WHAT server your client connectes to, the all connect to the same database. How does this work with game worlds? Not entirely sure. If you are managing by geography, then you have built in load balancing (assuming your audience is equally geographically spaced) and you limit logins to a particular server, shunting people off to another area if your server gets to full (reaches a critical threshold that the load balancing device is aware of) and do the world mapping that has been discussed in other threads.

In short, you have to do some "clustering" just for reliability reasons, but it doesn't have to be true parallel processing and you need to spend some coin to get the server farm up to snuff.

Now, the real question is: does this have anything to do with the thread ;)

#27 Wish I could edit the bloody post ;)

By the way, Oracle Development licenses are free. So you can develop you db without buying it. Then, find a host who already has it ;)

#28 Would like to mention that MySQL may be nice to start, but Postgre SQL offers better expanability because it's open source. I don't know the details, flame me all you want, but most Linux buffs (IE #linux on efnet) prefer PgSQL. Both offer a very nice set of features. Honestly I've used both and found MySQL easiar to get out of the box. Both have equivelent plugins for Apache, PHP, Perl, and Python. I like PgSQL's user management system better. And MySQL's C++ API forces you into this wierd OO programming scheme. If you're considering a database, be sure to play around with them first.

#29 Ed, your large post is VERY relevant to this discussion!

Many people have never completed a network game, and as such dont really understand the amount of "back end" that goes into a MMOG (and by this, I mean a central server model such as Asherons Call or Everquest or Ultima Online).

I think Oracle and Solaris is pretty much the route I would expect to take myself. But to be honest, I dont think that route is really a viable model much anymore, mainly because there are too many people already developing these things.

If you could live with not having total central control however, I think a distributed MMOG is much more viable. Having a whole bunch of smaller "cells" of servers, with about 100 people each, seems to be a much more usable objective (given that you could create a test platform with a single PC etc).

Distributed systems tend to scale a lot better for this style of thing I think, because it can be scaled to load. Obviously it changes the game design considerably though.

Phil.

#30 For anything larger than a proof of concept, yeah, Oracle/Solaris or W2k/SQL2k is the way to go.

Other things to consider when making this choice: Who maintains the datacenter after launch? Traditionally, Windows servers are a bit easier and less costly to maintain (people wise, again, equipment is equipment). SQL server DBA's are easier to find, NT troubleshooters are easier to find, everyone and their brother in IT has an MCSE ;)

Solaris and Oracle take a finer touch. Unless of course, after you code your game, you are going to act as sysadmin and DBA for the world you built :)

Shouldn't control your deployment, but it should influence it.

I would be interested if anyone has done large-scale implementations of either postgres or mySQL.

Now, a distributed system, that could be interesting. look at FSGS (http://www.fsgs.com), they have a distributed game matching service, everyone reports to a central database, but you can download the server and be a "node" in the worldwide game matching service (like Blizzards Realms servers for Diablo II).

My brain is grasping at something here, more coffee, but there is something to what you said about a distributed system like that.

#31 Off Topic

Edward you should be able to edit your post by clicking on the icon immediately to the right of the posted date. If not please send me e-mail and I will look into it.

--Rick

#32 I had a really bad time with NT4 and SQL Server, but I admit I'm not up on running that system (but I did kind of expect it to work "out of the box").

Distribution is (imho) the ONLY way a small developer can expect to garner enough bandwidth and server power to host a large multiplayer system.

You have to understand the whole realm of distributed services though, which isnt something many people have any knowledge of (especially in terms of hard core games programmers).

This is what I did my degree in, so I have some knowledge of the issues involved (not that I'm current anymore on these things, but the core concepts dont change), and it seems to me the best chance to be able to scale to a lot of players for most developers.

Imagine the whole worldwide web being hosted from a central server? it just wouldnt fly. Why would we expect games to be any better?

Phil.

#33 I Totally agree with Ed on one point and Totally disagree with Ed on another point.

First, I agree that using a Oracle or DB2 is much prefered for a large scale project then is MySQL or Postgress SQL, both of which I have extensive experience with. We currently use MySQL to drive our website and automated leagues and ladders, and it works very well for the current size, but MySQL and Postgres SQL are not long term solution for Major Projects like a back-end for a MMORPG. I would feel much more confortable with using Oracle or even DB2 as a back end, and both are free to develop if you are using linux as a back end.

Second, I totally disagree with the commnent about running a NT/2000 solution is less expensive, if you look at the total spectrum of costs of software, licenses, hardware, and staff I still boast that a unix/linux solution is more "cost" effective.

#1) From a software and Licenses perspective, both unix and Linux are more cost effective then Win NT or 2k, though this is not big enough a difference to make a decision on to be honest.

#2) Hardware will be comparable, cause as ed said for the most part hardware is hardware, and actually hardware for a unix system runs a bit more depending on the configuration.

3) (the big difference in my mind and experience) Here is where I disagree with Ed, and take it for what you want. Because of the Stability of Unix/Linux systems in the long run they are less expensive to maintain from a systems standpoint.

Why? since Windows is not as stable (and I am sure I will open up a can of worms here) you actally need more employees to maintain the systems, so where I may spend 80,000 per year of a good Unix Administrator, I spend 60,000 per employee for 3 NT admins. Do the math..

How do I know this? well My IT department went from 2 employees and 2 contractors (fulltime) to maintain our NT based Network, and now we have 1 employee and 1 contractor (6 hours a week)with our Linux based network. Plus I have done several other implementations where the IT staff was reduced dramatically by applying a Unix/Linux solution.

Other problem with NT/2000 people, they are not as good at trouble shooting for the most part, as a UNIX person. Now dont get me wrong I know some really really good NT/2000 admins, but as Ed says everyone and his brother is a NT person and unfortunately alot are not very good at trouble shooting (outside of rebooting the machine)

Why do I think a Unix admin is better at trouble shooting? Simple, most the time a Unix reboot will not fix your problems (actually a windows reboot doesnt either in most cases, it just resets the machine) A Unix/linux admin must know how to fix the problem, and can do so using the command line which allows him to actually see what is wrong and how to fix it.

No offense I hope Ed, cause you make alot of good points, I just happen to disagree with you on that one :)

#34 None taken :)

I have deployed both large Solaris implementations and large NT implementtations. Usually, it's dictated by the experience that the client has, and any investment they already have in infrastructure.

But, it's just easier to show someone how to "do it" in the Windows world ;)

I actually prefer Solaris :)

Yes, Windows has stability issues. Uptime on a windows box is frequently much less that that of an NT box. But, if the application is well written, and you have decent failover, you pretty much even the field.

Now, we might be ranging afield from the topic :)

Solaris for Intel is pretty much free nowadays. Oracle development licenses are free. You can build the system and beta it without spending a dime on the OS and the software. Given that, I still might go either way. Look at the requirements, the programming talent you have available, and project some administration budgets :)

Then make the platform decision.

Now, where was I on that distributed systems musing...

-edit-
THAT'S how you do it, must have been blind, thanks for the tip.

#35 I would say don't go win2k. The issue that isn't talked about often is post-development. A network operations group is going to be supporting these beasts. If you have a server farm which includes hundreds of machines darkside control is key. Win2k definitely adds a lot (compared to NT) in terms of headless operation but there are still operations that require an operator to be sitting at the box. I think any unix variant will supply an admin with more cli options.

I have a question about the NICinterrupts issue - Has anyone played with multiple procs and setting processor affinity for a particular NIC? Win2k supports this and it seems like this would get around the interrupt issues - one proc services interrupts from the NIC and the other just chugs away on data.

#36 My $0.02 on the database issue, if I might suggest, people might be interested in looking at an Object Oriented Database called ObjectStore from http://www.ObjectDesign.com.
They have been around awhile and deliver on multiple platforms, including Linux.
Disclaimer: I work for them as my day job.

#37 I don't know if I would call TGE "V12".. isn't that a bit outdated term considering all the changes / improvements that have gone into the product. Sorry to nit pick.

#38 The last post on this thread was 2001.

It was V12 back then.

#39 Lol. sorry. thats just kind of funny

#40 There is a group who has successfully got a framework up and running for a MMO with Torque www.graalonline.com they have melded it with their current technology they use for their 2d engine but I have seen it and it does work. But yeah LOL this post is like one of the first I ever made :)