Research: (M)MOGs system for Torque...

#1 Interesting... a libary that can be used for both T2D, TGE, and TSE for making (M)MOGs? But I think TGE and TSE already have the MOG part done pretty well...

#2 Yes, non-persistent MOGames can be done with Torque out-of-box, we are proposing persistent MOG and, mostly, MMOGames without zoning, e.g. all the players can "see" each other in continuous game world.

#3 This sounds interesting for sure ^^

#4 You're going to take out the network code from TGE/TSE/T2D and then roll your own as well as implement the other changes proposed? Sounds like a huge undertaking.

Quote:
* The system will be stable with 100,000 concurrent users and more (warning: spoiler :); this hardly depends on provided server software/internet pipe/game type, but we are crunching 10,000 concurrent users (simple arcade T2D game, all players are driven by AI scripts) on Pentium2/64Mb RAM/5 Gb HDD/CentOS 3.4 Server/3Mbit Internet machines, viva optimized kernel, Assembler and C :))

No way on earth that you're gonna do that. :)

#5 2Stefan: no we are not going to take Torque networking code for server, it does not support big amount of connections per second, it was just written that way.

> No way on earth that you're gonna do that. :)

Why not? Amazon.com serves that amount of concurrent users and in tech words, it's the same as running a game server - in both ends they are running scripts...
We'll see it on Open Beta ;) The only problem is to find so many testers and funds to buy this amount of the bandwidth. We can simulate it in our lab using Gigabit network cards, but our internet pipe is only 5Mbit wide...

#6 No offense but I do agree with Stefan. I dont see how you will be able to create a system that will be stable atleast in a realtime enviroment for 100k concurrent users.

There are ALOT of differences between Amazon.com and a MMO Game server. Amazon doesnt have to deal with collision detection, interaction between objects, AI, ect.

"Amazon.com serves that amount of concurrent users and in tech words, it's the same as running a game server - in both ends they are running scripts..."

The first part of that might be true. But the second half of that is very far from being true. A website serveing data with simple connections to a database is VERY far from a MMO game. The website doesnt have to deal with the crunching a server would have to deal with. (AI, Collision Detection, Ect.)

It would not be hard to make a server that does nothing but accept 100k connections. But for the server to be even remotely secure there is ALOT more to it.

Also "but we are crunching 10,000 concurrent users (simple arcade T2D game, all players are driven by AI scripts) on Pentium2/64Mb RAM/5 Gb HDD/CentOS 3.4 Server/3Mbit Internet machines" wont be able to handle a 3D Enviroment. Maybe a 2d System where options are alot more limited.

Ageain though this is just from my own experience. If you can pull off what you say (though I do doubt it) you will be able to achive what many major budget companys have yet to be able to do.

#7 For amazon (or any http server) a page request can take between 1-5 seconds (or more depending on connection speed).
Web pages aren't realtime (for the most part) and aren't expected to be by most users.

So not really the same thing.

~neo

#8 Tom and Neo:
I do agree and disagree.
I'm not talking about ONE server ;), I was talking about A SYSTEM. (Just FYI, I'm older then 25, have no pink dreams about creating my own MMORPG ;), and have a good exp in coding heavy loaded systems). Amazon written in Java and it DOES compute alot to create recomendations, analyze user choises, etc. Collision and AI - just few formulars that are crunched in 10-20 msecs for each avatar. 3d dimention just adds some extra variables to the whole thing. If you split a game tick from the Torque demo - it's like serving hundreds web pages, 2-20 msecs each. And most of the AI is inside a script.

BTW, what about EVE-Online space MMORPG? They can technicaly serve 100,000 concurrent players, they just dont have enough of them to be online at the same time ;)

> So not really the same thing.

Sorry, but it is the same thing in a big scale.

>If you can pull off what you say (though I do doubt it) you
> will be able to achive what many major budget companys
> have yet to be able to do.

Who said that they have good ideas to do such a thing? Have you seen their developing process from inside? I did. Mostly, they skip the big UML/graph analysis step and jump right to the coding of a demo. Big mistake. That's why I do have respect for CCPGames (Eve-Online) and I dont for Everquest II-like, etc. MMOGs (sorry guys, the game maybe interesting, but the implementation is awful...)

#9 Hehehe, alex I'm not trying to dispute the issue with you, I merely raised a point
re. amazon et al. not being the same as mmo's in the sense that mmo's have to
simulate EACH and every client at say 20ms each and amazon is not limited to
that time slicing for realtime feedback. So one would have to have more servers
in proportion to the processing required per time slice.

For example where one would need say 1 server to process 10000 requests
over 5 seconds on amazon one would need 10 servers to serve the same amount
of requests in 500ms.

That's ALL that i was saying.

I have a background in large distributed systems and so yes I do know what you
are talking about.

In essence one could have a billion users if one wanted by just throwing more
hardware at it. Whether current infrastructures (and one's pockets) allow that
is another question ;p

I wish you success in your endeavours.

~neo

[edit: grammar and server count ammended ;p]

#10 Web sites also have caching and search prediction, regional servers, and possibly use services like Akamai. I'm curious about hard data... with 9,999 AI users what the 10,000th users gaming experience is like? Is the memory pegged out? 5GB how much is taken up by the page file?

Win2k requires more then 64MB RAM... technically you can try to run it with less but you'd be crazy to try. Might update the minimum spec's on that point.

I'm not saying I don't believe, I just want some more specific data because it does sound too good to be true.

#11 I just wanted to pop in and list some of the specs required to run an MMO, per user:
There needs to be at the very least, 5kb of ram per user / object. (Physics, even gravity, need to be handled server side after all)
As well as a bare minimum of 1ghz per 500 users (Assuming you have collision detection with all objects, have each object and terrain loaded into server memory, etc.) up to about 2,000 users; past that is anyone's guess.
As well as at LEAST 2-3kb/sec per user.

With that in mind, it would take:
(Keep in mind I also slightly factored in some basic things; like loading the world into memory - but its a relatively small world)
10GHz+
5gb ram (JUST for players, probably around 8-10gb with objects)
and a 2.4Gbit line


Running an MMO server is a lot different than running a webserver; a webserver is called into memory, sends the data requested, then unloads. An MMO server is -always- running, -always- keeps -everything- in memory, and makes many calls to the filesystem or database every second.

Also keep in mind that no one complains if a webserver takes a few seconds to send data, because its not noticeable. However, in an MMO, or any online game - players must receive and send data to and from the server with under 250ms latency, otherwise lag becomes a major problem.

Keeping the latency low means you have to have a 10-15% overhead of the basic network/system I laid out above... The prices involved in such a machine and such a network, are so insane its just not worth it.


However, if you are referring to a complex system of networked computers handling certain areas of the game, its completely different; while you still do need that overall network speed, the system specs themselves are drastically lowered.

All in all: MMOs generally only accept 2-5 thousand players for a reason.
(All of the above are assuming the game is 3d; the specs for 2d games are drastically lower, but still relatively high. The network speed is also quite a bit higher if all players are real players (As the server has to send and receive data to each client)


----------
Enough with my ranting, I don't mean to burst your bubble or lower excitement of your project, but I wanted to make sure you understood what goes into a real MMO engine ;)
(I get a bit worked up when I see people claiming their network engine can handle 100,000 concurrent users; as such a task would require technology (3Gbit line) not fully available to the public (For under $100,000/month :p))
Back onto the original subject:

If you manage to release such a library, I wouldn't mind seeing it in action; while Torque's network library is one of the best I've seen, it'd be nice to see what an indie crew put together :)

#12 Maybe a cluster of servers may work better? Maybe use these cluster servers to "regulate" data, while data is send P2P?


Edit: SP?

#13 MMOG Middleware Pack for T2D, TGE and TSE.
Bigworld... without the million dollar price tag ...

:)

#14 @Neo: You got me :) I was just teasing. That's right, an indie developer can use 3-5 cheap machines collocated somewhere. It's quite a reliable solution.
My mission is to prove that better doesnt mean bigger (following Microsoft consept in using computer resources for new versions of their software, but I stop here to prevent a flame war ;)).

> I wish you success in your endeavours.

Thank you!

@ Jon Ferris: We have 200 Mb of swap and 30Mb of it is used. 56Mb of RAM is used, 4Mb stays free. Yes it does sound too good to be true for me too, that's why I decided to prove that the old school is still alive :P This is for a Linux box, Win box will need triple of that size.

> with 9,999 AI users what the 10,000th users gaming experience is like?

You'll have an opportunity to check that in a year or so (open beta) ;)

@ Sykoii: You did the right math. Just few corrections: CPU of a server is not always used by all of the players, so a good written server for 2-3,000 players will "eat" 45-60% of the CPU time. 5Kb per player can be reduced too, if a developer will not obuse OOP programming principles (using classes a lot). In a 3D shooter user may need about 0.5Kb to store his status/position/etc. The AI code can be coded as a state machine (one class object for all the avatars of the same type, e.g. goblins) and use data from a player structure, e.g. it will be "data driven", not "code driven". Having an exp in coding of a cell phone game helps a lot to optimize code and make it "data driven" in a good way, because of the device constrains (64Kb for game code and data, 10-60MHz CPU for AI and drawings).
And you may need 0.2Kb-0.5Kb/sec per user (plus chat text, can be compressed too) if you are in the classic MMORPG business.

> An MMO server is -always- running, -always- keeps
> -everything- in memory, and makes many calls to the
> filesystem or database every second.

A web server does the same (with light diffs). ;) A web server under heavy loads does not unload/load itself (it's too expensive), it just pools the respond/request objects in memory. The reason it takes 1-2 seconds to receive the page is inside HTTP protocol (establishing a connection every time to fetch a page (expensive!), sending plain text over net, which should be parsed by a browser). Try to create an UDP-based webbrowser sending binary compressed data: you will get your page in 100-250 ms.

> but I wanted to make sure you understood what goes
> into a real MMO engine ;)

I do :P

> I don't mean to burst your bubble or lower
> excitement of your project

Thank you. We all need to dream sometimes to motivate ourselves, right? :)

@ Chris "DiGi": yep, can be a solution
@ Randy "Vashner": That's the idea ;)

#15 I've got some interesting emails, people are really interested in the project. Thank you for all the emails, guys!

#16 Someone above here seriously underestimates the need for cpu power.

8 Nodes (servers) consisting of 8 AMD Sempron 800 series cpu's (dual core) for a total of 16 core's per node and about 64GB of memory at the least ot be able to serve anywhere near 100k users. So, you'll probably have space for about 200 zones or so. (TGE processes interconnected with trigged events to connect to next zone etc). Now if you manage some multithreading...

You could of course use other systems, like a Sparc based one. A system like the one outlined vaguely up there would set you back somewhere between 200-250k USD just in the actual equipment. Other costs might apply as well.

And you'd need something like a OC-12 pipe (655Mbit) and that only gives you 0.81kB/s per user.

This should work as a base. So depending on your load/mission/scenario you would either scale back or upgrade.

If I knew more about the estimated software resource usage I'd be happy to draw up some hardware requirements. But in general I have a feeling it's being very underestimated.

Also another thing you *have* to consider. If you make a mmo system it's not going to be worth anything if you don't include the support side of it. Tools to handle log data, player petitions, GM access levels, bug reports etc..

#17 Alex: I've done pretty heavy testing with a game of mine in development; with only 10 people in, and no network compression (Adds to CPU usage just a small bit; you can't use high compression either, as the data needs to send very quickly) it takes 25-30kb/sec (Just for basic 3d movement), and a few % of the CPU (Just a crappy 1.7 celeron).
As for your webserver comment: I've run numerous webservers that serve tens of thousands a day, and trust me when I say they do not even come close to the power required by an MMO (Both network and CPU/Ram usage)


Anders: Those were actually my original specs, but I decided they were a bit too outlandish and changed my math a bit... Theres no way to tell how much processing power / ram it would take past 2,000 or so users, as its just not something that has been done in a real world scenario. Different rules apply to the required amount past 1,500 users, so its hard to make an estimate not knowing every step up.
(Different rules apply at 1,500 users because that also sort of factors in how much memory it takes to load in the world, objects, etc.)

#18 I agree. It'd be smart to try connect different zones of a world together, running on different processes, so that you keep the real simultaneous number of players on one actual process down to below 500. If you can do this without the player noticing too much, you have a good design. In MMO's the load balancing also becomes part of the actual game world design in that case. So making sure players have a reason to spread out could be something to think about. It also help you spread the load out on several boxes (cheap) than having it all happen on one big box (expensive)

#19 Andres, those systems described would only be used if there actually was a demand for them(IE: having 50-100 000 people online continually) which is getting into the realm of Everquest or Final Fantasy online populations(around 500 000 subscribers = an average of 100 000 or less online users). With an indie game you can expect populations of much much less than 1/20th of that if not a lot less. Something that a group of pentium 4's would handle just fine.

Also 1/2 a decade of hosting/server experience has taught me never ever use AMD's for something you need to have up with 99.9% uptime as it will not happen. More bang for you buck without a doubt, however you sacrifice that for stability when you are dealing in a server loads over 0.5 .

AMD = good for playing games
Intel = good for serving games/websites/etc etc.

#20 That might be true 6-7 years ago. Granted, I wouldn't use an XP or MP system, but their Opteron IS of enterprise quality. The only question is, is the software running on it of that quality too? ;)

And if I would take your view on these things, then I would certainly trash both AMD and Intel and go for the real deal, UltraSPARC, Itanium2 (Alpha) or IBM's POWER processors on AIX.'

I had over 600 days uptime on a dual MP2000+ game server a couple years back, until we replaced it.
Can't say the same for the applications running on it. But the Linux OS we used on it was rock solid.

Edit : Fixes ;)