Product not found

Game Development Community

Mythos

by Prairie Games · 05/20/2006 (1:37 am) · 12 comments

MYTHOS

Mythos is the codename for the next generation of Prairie Games, Inc's technology. I have been researching components and formulating a plan for their integration. It's been fun playing this iteration of connect the dots. There are themes emerging and I dare say some standards. It should actually be pleasant working with some of this stuff!!!

A big change in our technology will be the dropping of single player support. This will have a dramatic impact on the "scaleability" of our multiplayer code. I'll be able to make decisions about servers supporting hundreds or thousands of players without constantly bonking my head into the embedded single player server. The client will also be "clean".

I will first concentrate on the editing systems and content pipelines. I would like to see most of these systems available for Torque licensees and others to use in their own projects. I feel it's important to strengthen them with testing and usage. I also plan on publishing the source code to the new and improved Torque python support. The runtime system, which includes TSE, will be under a proprietary license. It's possible that the runtime (or portions thereof) will be licenseable. There is a fair amount of time for us to consider this... I have about a year to complete this task. I will also have other responsibilities over this year, so it will be more or less a sprint. What's new?

I wrote some notes for our company and I thought they may be interesting or useful to others. So, I dressed them up a little and am publishing them here. The notes follow.

-Josh Ritter
Prairie Games, Inc

MYTHOS RUNTIME

Component: Torque Shader Engine

License: Proprietary

Description: The Torque Shader Engine is an evolution of the Torque Game Engine which has it's roots in the Dynamix game Tribes2. It's focus is in code cleanup (see 1.4 merge) and replacing the aging TGE graphics subsystem.

Usage: TSE provides us with UDP based simulation networking, a GUI system, an integrated (realtime) world editor, a large scale terrain system, platform services, and a console system that exposes convenient hooks.

Issues: The TSE is currently at Milestone 3 and in "early adopter" release. As such, there are a number of bugs and missing features. TSE's collision and physics systems need an overhaul both in terms of features and performance. The current scenegraph (replaced in MS4?) is also inefficient and convoluted. The DTS and DIF formats both have severe limitations and are unwieldy for tools. Datablocks aren't a good way for us to store and transmit game configuation data.

Requirements: The TSE console system has been exposed to Python via the pytse module. We will replace the collision system and add a new physics system. The physics will mostly be client side and minimal networked consistency will be maintained. We will wait and see what MS4 brings in terms of scenegraph improvements and lighting. Though, we may implement an octree with efficient mesh storage in the meantime. Datablocks will either be removed or made to load locally on the client/server. We need to evaulate what to do with Shapebase and Player classes (perhaps write something new based on Gamebase?). I would also like to strip out as much dead code as easily possible, things like the old TGE terrain system.

Source Code: Available for licensing at http://www.garagegames.com/products/28

Component: Bullet Continuous Collision Detection and Physics Library

License: ZLib License

Description: Physics library written by Erwin Coumans a previous Havok and Blender employee

Usage: Bullet will be used for collision detection and game dynamics. The Blender Game Engine uses Bullet for simulation and thus provides us with a good content creation and testing environment. An interface between Bullet and Collada is being generated which will be useful in the tools chain. Bullet also has binding to ODE. We really need to improve the speed and flexibility of TSE's collision system. We also want some dynamic objects to interact with in the game world, perhaps including ragdolls. It's important for performance and implementation time to make this as network independent as possible. We don't need a 100% accurate networked physics simulation. If we did, the programming work and server side CPU involved would be preemptive. The physics work should be handled on the client, anomalies being acceptable for our gameplay.

Issues: Replacing TSE's collision system is non-trivial and will involve significant work. There will be changes necessary to the world editor and scene lighting. We will also need to make it work with Atlas and generate physics information for whatever mesh representation (besides, DIF/DTS) we happen to use or generate.

Requirements: Other than the TSE work, we need to generate a content pipeline which includes attaching the physics information to our mesh data. The Collada 1.4 exporter/importer for Blender has some physics support. I need to look into what serialization support the Bullet library has currently in terms of the Blender Game Engine and in the Collada interface being generated.

Source Code: http://bullet.cvs.sourceforge.net/bullet/bullet

Component: Stackless Python

License: Python License

Description: Stackless Python is an enhanced version of the Python programming language. It allows programmers to reap the benefits of thread-based programming without the performance and complexity problems associated with conventional threads. It has proven extremely scaleable in the EVE Online MMORPG technology. A presentation of Stackless Python's usage in EVE Online is available here: http://www.stackless.com/Members/rmtew/code/PyCon2006-StacklessEvePresentation.z...

Usage: It's very important that scriptors can generate code for both the client and server in (relatively) anonymous functions. It's also critical they don't have to manage a lot of bookkeeping code for performance and integration. Stackless's tasklets and channels are a very good fit for exactly this... they have a very natural feel when compared to python generator functions. It's cool that something as mind melting as coroutines can actually make it easier for non-programmers to write code!!!

Issues: None at this time

Requirements: I need to look at breaking out the pytse main loop further using tasklets on both the client and server side. The system that uses this also has to be designed and implemented.

Source Code: http://svn.python.org/view/stackless

Component: Twisted

License: MIT License

Description: Twisted is an event-driven networking framework written in Python. Twisted projects variously support TCP, UDP, SSL/TLS, multicast, Unix sockets, a large number of protocols
(including HTTP, NNTP, IMAP, SSH, IRC, FTP, and others).

Usage: Twisted is extremely useful for writing client and server code very, very quickly. It will be used for our Master Server, Central Character Storage, Region Servers, and also on the client. Minions of Mirth uses Twisted extensively.

Issues: None at this time

Requirements: I need to investigate the new perspective broker which was recently released in preview form.

Source Code: http://twistedmatrix.com/trac/browser/trunk

Component: Axiom

License: MIT License

Description: Axiom is an object database, or alternatively, an object-relational mapper.

Issues: None at this time

Usage: Axiom strikes a balance between OODB and RDBMS models. It uses SQLite as a backend, which simplifies database access, configuration, and deployment. SQLite also is an amazingly fast database engine.

Requirements: Client and server side databases, included shared data. There will be a system using Axiom to augment or replace TSE datablocks.

Source Code: http://divmod.org/trac/browser/trunk/Axiom

MYTHOS TOOLS

Component: Pedestal

License: TBA

Description: Pedestal is our integrated RPG editor will allow realtime editing of the ruleset on live servers. It's GUI will be based on TSE native controls with the bulk of programming being in Python. Pedestal data is stored in Axiom data storages which can be plugged and unplugged. Additionally, all data/objects receive URI addresses and can be inherited or overridden in the same or external data storage. For editing purposes, data storges can be passed around and then merged back into the main storage with conflict resolution.

Usage: Editing of role-playing data and rules, with realtime feedback on changes. Also, versioned import and export of XML data from Calc, Open Office's spreadsheet.

Issues: Non-existant

Requirements: Pedestal requires the pytse module and the new pluggable data storage system to be implemented. There needs to be a "generic" editing interface for data similar to the TSE Inspector. This interface should be easy to override for custom cases. The region server needs to be operational for realtime editing.

Source Code: TBD

Component: FCollada and Collada

License: FCollada - MIT License, Collada - SCEA Shared Source License

Description: COLLADA is a COLLAborative Design Activity for establishing an interchange file format for interactive 3D applications. Originally established by Sony as the official format for PlayStation 3 and PlayStation Portable development, COLLADA continues to evolve through the efforts of the Khronos contributors. Dozens of commercial game studios and game engines have adopted the standard. FCollada is a C++ library which offers incomparable support for COLLADA interoperability.

Usage: Provide a common data format for geometry and physics processing.

Requirements: Where we are interfacing with Collada with no existing support, a decision has to be made whether to go with Collada sources or use the FCollada library. We will likely generate a new unified class/mesh format for TSE that puts a premium on efficiency and flexibility. It would also include physics and collision information. This format would have a sole processing point from Collada. This may end up being a Collada -> DTS convertor after all, due to the Torque Show Tool. This somewhat depends on if we can republish the Torque DTS Plus SDK source code.

Source Code: FCollada, Collada

Component: GTKRadiant

License: GPL

Description: Excellent CSG based editor for efficient modeling and texturing of architectural models. It features bezier curves, plugin system, and a shader scripting system which is integrated well with preprocess tools. GTKRadiant can be freely redistributed.

Usage: Architectural, brush based, model generation.

Issues: None at this time.

Requirements: We need to generate a small shader library specific to our needs and make a "Mythos" game plugin. This should be quite easy.

Source Code: https://zerowing.idsoftware.com/svn/radiant/GtkRadiant

Component: Q3Map2

License: GPL

Description: CSG pre-processing tool with excellent radiosity, PVS generation, and a shader system which can guide computations. It also has "meta-brush" support which significally lowers
polygon counts of brush based models.

Usage: Processing of .map files exported from GTKRadiant (or other CSG tools).

Issues: None at this time

Requirements: We need to create a "NULL" shader which will be used to mark surfaces which will be discarded. Also, we need to come up with a way to mark interior and exterior surfaces. The PVS should be collapsed into much larger contiguous zones for batching purposes. We need to write an interface to Collada and thus the mesh library. Also, I would like to be able to import these models into Blender including lightmap with texture coordinates. The Blender Game Engine supports a second uv coordinate, though I am not sure what the import functionality requires.

Source Code: https://zerowing.idsoftware.com/svn/radiant/GtkRadiant/trunk/tools/quake3/q3map2

Component: Blender

License: GPL

Description: Blender is an excellent 3d modeling and animation package. It features an integrated "Game Engine" with realtime physics editing via the Bullet dynamics library. Blender is fully scriptable via Python, as is it's "Game Engine" functionality. Blender's interface has gone through some major revisions in the last years. It is now quite intuitive and polished. Check out Elephant's Dream for a great example of what Blender is capable of...

Issues: None at this time

Requirements: We will focus on Collada export from Blender, so using 3dsmax/Maya will remain a possibility thru their Collada based exporters. We must generate working example content files for characters and other models, with physics information. The export and test cycle must have a fast turn around. Here's a link to the Blender Collada exporter and importer plugins.

Source Code: http://projects.blender.org/viewcvs/viewcvs.cgi/?cvsroot=bf-blender

Component: Trac

License: BSD

Description: Trac is an enhanced wiki and issue tracking system for software development projects.

Issues: The last I checked, the subversion repository being served has to reside on the same local box as the Trac installation. It would be nice if it could be accessed remotely, though this isn't a requirement.

Requirements: I have to setup the subversion repository for our public code and also setup the Trac installation on this box too. I need to lay out the repository, and it's many dependencies, as intelligently as possible.

Source Code: http://projects.edgewall.com/trac/browser/trunk

Component: Subversion

License: BSD

Description: Subversion is used both for our development versioning and our end user patching system. It's great and handles binary files very well.

Issues: None at this time.

Requirements: I want to keep compressed files in the (game) end user repository to decrease bandwidth usage. We'll continue to use generated sha checksums and local/remote caches manifests.

Source Code: http://svn.collab.net/repos/svn/trunk

Component: Stani's Python Editor

License: GPL

Description: A great Python editor with integrated Blender support.

Issues: A few small bugs, nothing major

Requirements: None

Source Code: http://svn.berlios.de/wsvn/python/spe/trunk/_spe/?rev=0&sc=0

Component: Large 3D Terrain Generator

License: Proprietary

Description: A very good tool for generating large scale procedural landscapes.

Issues: There may be some issues once Atlas has realtime editing, texture painting, and lighting support. L3DT is really useful to "baseline" a large amount of terrain. It does provide some hand editing, but for the most part it's quite procedural, which isn't all bad.

Requirements: We'll know more once Atlas is further along.

Source Code: NA

#1
05/20/2006 (2:26 am)
Pretty neat read. Glad to see that a game company in my home state has a bright future!

Thanks for the links too!
#2
05/20/2006 (3:33 am)
Its great to see where you stand on the pipeline side of things. Especially with Collada and wanting to fix the dts/dif. Im pretty much in the same boat with you even though my pipeline is Maya based. Much luck to you.
#3
05/20/2006 (4:13 am)
I haven't completely discounted 3dsmax/Maya, though I really want to give Blender an honest go.
#4
05/20/2006 (4:16 am)
Hey Josh,

Excellent read. Thanks for laying out your thoughts for all to see. It also helps me see where we might need some more tools for Torque... :o)

- LightWave Dave
#5
05/20/2006 (7:47 am)
This next step u are taking sound ausome, abig leap than MOM :)

congrats on your new direction Josh
#6
05/20/2006 (11:02 am)
I kinda wish I didnt absolutely gag every time I see any form of Python because I'm absolutely in favour of using proven solutions to problems, especially any high quality open source ones.

Having said that, I've heard mixed reactions to Collada. I'm going to have to study thier ethos in a few weeks time to see if thats not something we might want too.

I agree about the coldet and physics, its just not stable enough for many circumstances. Having said that, for your purposes I'm not sure how far you need to go there either. I'm looking at Physx integration, figuring that server hardware with physics hardware might actually mitigate some of the client side processing.

I like how you've broken things down though, its probably a very wise move if you plan on continuing down a specific path.

Have you considered FBX format btw? Its widely supported by tools and is now available from Discreet/Alias
#7
05/20/2006 (11:11 am)
Well ive discounted Max. :) haha. j/k. Yeah packages are pretty much about personal preference. I find Elephants Dream interesting though. Maya is just a personal preference as per its the easiest for me to pipeline and I know the Mental Ray connection really well since I helped develop it. Either way it shouldnt make a difference with Colloda support. Truthfully as an indie I should look at XSI but I have to look at whats more important to me. The extra money cost in license or the time it would take me to pipeline out XSI as well as I have Maya right now.

Either way im glad the way you are viewing content generation and how important quick iterations are. I miss what I had at one shop of being able to build what you wanted in Maya(be it characters, cinematics, or environments) and having it show up and work in engine easily. The less I have to struggle with asset generation and getting into the engine means I can devote more time to making it look nice and overall gameplay.
#8
05/20/2006 (11:17 am)
Phil:
Ive used FBX in production. Ive had issues going from Max to Maya and vice versa while I was working with a very large amount of assets and cameras were an issue if I remember correctly. The filmbox format was mainly done to facilitate the exchange of filmbox motion data between packages. So for characters between things like motion builder and other products it works pretty well. I just wouldnt trust it for much else.
#9
05/20/2006 (4:43 pm)
@Josh,

I am curious about you choosing Collada at this time Josh since the whole format is still being developed and worked on at this time so there is always that issue of what it can become. It certainly has the potential to be a useful format, but it also has the potential to get pretty bloated too depending on how much support they begin to try to add in to appease VFX guys and convince them to use the format too.

@Juan,

I am curious to see what will happen with FBX now that Autodesk owns it, Maya and Max. I am expecting some pretty good things to come out of it in the very near future.

Edit: You are 100% right though that right now its a bitch to use FBX to move data around. The running joke is that you have to make a pagan sacrifice or two to appease the FBX gods in order to get things working.

PS: The Collada exporters for Max & Maya actually use the FBX routines, how's that for irony :)
#10
05/20/2006 (4:59 pm)
Quote:A big change in our technology will be the dropping of single player support.

Interesting, as MoM started as a "Massively Single Player RPG".

Funny, life's twists and turns.

Good luck, Josh! =)
#11
05/20/2006 (7:12 pm)
I am looking at the FCollada source code for it's MAX/Maya exporters and not seeing any reference to FBX.

I am using the Collada format to integrate our tool chain with custom processors. The libraries available for reading and writing the format are quite nice. The format is a good fit for game assets.
#12
06/03/2006 (5:50 pm)
@Josh

I'm actually quite interested in using Python with Torque. My only question is, how do you protect your source scripts?