More Uses for XML in Torque
by Derik Wilson · in Torque Game Engine · 03/08/2009 (6:49 pm) · 7 replies
XML is a very standard and structured vehicle for sending, receiving, and formatting data in pretty much any system. How can this help game developers using the Torque Game Engine?
You may already be familiar with XML in its more traditional context, sending and receiving data over the web, streaming content from your web server, sending messages to your mobile devices, etc. What about using XML in your game? What purpose could XML possibly serve in your game? In my opinion, XML isn't just for transmitting data from system to system, it is also a very elegant way to send and receive data from component to component within a system.
As far as I know, since Torque Script has been in existence, there was never a viable way to return arrays from functions until James Ford enhanced an older resource by writing the TscriptArray class. This still leaves a lot of coding for the programmer if they want a more structured way of storing data.
I had an idea for a more abstract approach to storing and passing data between functions in Torque Script. That idea involved integrating XMLite into TGEA 1.7.1, and writing a wrapper that would abstract XMLite and make it a bit easier to get to from Torque Script. Since the game engine developers are called “Garage Games”, I decided to put a fun twist in the name of my project, “Basement XML”. That name actually comes from a running joke between me and the developers at Heaven's Blessings Tiny Zoo (the makers of Chariots).
I actually began brainstorming ideas when I came to the realization that it is sometimes very difficult to return result sets from a database query to the calling function without a lot of excess coding on the script side. So, I decided to put XML style tags around my column data and the tags themselves are the column names. This worked out perfectly, except that means I would need a way to parse it. I tried AsmXML, but it offered me no way to get the names of the columns so that I could grab data by column names. I ended up trying out XMLite, and I love it. It was very easy to learn and pretty simple to use. That got me thinking of how I would implement such a gem.
It's one thing thing to store a result set from a database query, but quite another to store 100 result sets from a database (or 1000, or 1,000,000). My idea was to create a way for the game server to only need to query the database once. If the query had been made previously, and it could possibly use the data again real soon, then the data would still be in memory in a structured format that would be easy for bxml to grab. The problem arises when there are multiple queries from the game server. They may be completely different queries, but either way, I don't want to crush the query that the getWeaponStats function submitted with the results from another query from another function. This got me thinking of multithreading. Since I don't know much about multithreading, I may just end up assigning unique ID's so that bxml will know where to get the data when needed.
This forum thread is set up to gather ideas from other folks in the GG community. So, fire away! I really think this would be a good resource; I just need to get the mind thinking outside the box, but that's difficult to do with just one mind. It's a lot better when you get multiple minds processing the data and coming up with conclusions (multibraining?)
You may already be familiar with XML in its more traditional context, sending and receiving data over the web, streaming content from your web server, sending messages to your mobile devices, etc. What about using XML in your game? What purpose could XML possibly serve in your game? In my opinion, XML isn't just for transmitting data from system to system, it is also a very elegant way to send and receive data from component to component within a system.
As far as I know, since Torque Script has been in existence, there was never a viable way to return arrays from functions until James Ford enhanced an older resource by writing the TscriptArray class. This still leaves a lot of coding for the programmer if they want a more structured way of storing data.
I had an idea for a more abstract approach to storing and passing data between functions in Torque Script. That idea involved integrating XMLite into TGEA 1.7.1, and writing a wrapper that would abstract XMLite and make it a bit easier to get to from Torque Script. Since the game engine developers are called “Garage Games”, I decided to put a fun twist in the name of my project, “Basement XML”. That name actually comes from a running joke between me and the developers at Heaven's Blessings Tiny Zoo (the makers of Chariots).
I actually began brainstorming ideas when I came to the realization that it is sometimes very difficult to return result sets from a database query to the calling function without a lot of excess coding on the script side. So, I decided to put XML style tags around my column data and the tags themselves are the column names. This worked out perfectly, except that means I would need a way to parse it. I tried AsmXML, but it offered me no way to get the names of the columns so that I could grab data by column names. I ended up trying out XMLite, and I love it. It was very easy to learn and pretty simple to use. That got me thinking of how I would implement such a gem.
It's one thing thing to store a result set from a database query, but quite another to store 100 result sets from a database (or 1000, or 1,000,000). My idea was to create a way for the game server to only need to query the database once. If the query had been made previously, and it could possibly use the data again real soon, then the data would still be in memory in a structured format that would be easy for bxml to grab. The problem arises when there are multiple queries from the game server. They may be completely different queries, but either way, I don't want to crush the query that the getWeaponStats function submitted with the results from another query from another function. This got me thinking of multithreading. Since I don't know much about multithreading, I may just end up assigning unique ID's so that bxml will know where to get the data when needed.
This forum thread is set up to gather ideas from other folks in the GG community. So, fire away! I really think this would be a good resource; I just need to get the mind thinking outside the box, but that's difficult to do with just one mind. It's a lot better when you get multiple minds processing the data and coming up with conclusions (multibraining?)
#2
Usually the database is not on the game server in an MMO, and in torque script, there are some limitations with passing arrays of data. In the past, we had simple string parsers in the script. The parser would parse through short strings containing delimited text data, or we would have set up functions to query for specific data, which can end up being a LOT of very specific database code, as well as many many queries. Problems arise with the delimited text method when you get a mix of multiple types of data that could possibly contain the character you are using as a delimiter (even TABs). Then I thought about using the TscriptArray class, or maybe expanding it, but I would still need a more managed way to get and receive data from within the system or from outside the system. That's when I figured that the overhead of a small parser (with limited features of course, e.g. non-validating) would work for my purposes.
After some more thought, I might implement this for our chat server communication, building a queue of chat messages, and take another look at using another method of caching data from the database queries. Instead of receiving the data from a DB query as XML, I could just stick it in a struct. Even then, I would have to store that struct in a linked list of some sort for caching so that I could access it quickly. This would probably make more sense since the game engine itself is making the database query using interfaces to access the selected database server (right now I am just using MySQL for testing but it is set up to handle SQLite as well for single player games).
I don't know, maybe you are right about the bloat. Hmm. I will have to implement another idea bypassing any XML building/parsing and see if I can benchmark it. I am wide open to other ideas though, because I still don't have the logic in place to actually cache the data. The only thing I have in place is the XMLite parser which works very well so far. I think I will keep that in there for a small XML parser in case we decide to use it for the chat system (communication between the game server and chat server, or between the chat server and client).
This is why I wanted to post something on the forums. The GG community has always been good about jumping in and stopping the foolishness before it costs someone an arm or a leg. ;)
03/09/2009 (3:21 am)
Well, I was thinking more of a generic solution for caching interactions between database and torque script (that's my current goal), but I also wanted to make it generic enough to allow for other, non-database, application. I am not talking about using the parser every time I want to query information stored in memory; my thought was use the parser to extract the data from XML and store it in a struct, and then store the struct objects in a list where I could access the information at any time until I don't need it anymore. But now that you mention it, yeah I suppose it was a silly idea since I can just use a struct, and build a list for caching the data without having to first format XML and then parse it later. I suppose the frustrations with Torque Script and trying to be generic made my mind run-a-muck.Usually the database is not on the game server in an MMO, and in torque script, there are some limitations with passing arrays of data. In the past, we had simple string parsers in the script. The parser would parse through short strings containing delimited text data, or we would have set up functions to query for specific data, which can end up being a LOT of very specific database code, as well as many many queries. Problems arise with the delimited text method when you get a mix of multiple types of data that could possibly contain the character you are using as a delimiter (even TABs). Then I thought about using the TscriptArray class, or maybe expanding it, but I would still need a more managed way to get and receive data from within the system or from outside the system. That's when I figured that the overhead of a small parser (with limited features of course, e.g. non-validating) would work for my purposes.
After some more thought, I might implement this for our chat server communication, building a queue of chat messages, and take another look at using another method of caching data from the database queries. Instead of receiving the data from a DB query as XML, I could just stick it in a struct. Even then, I would have to store that struct in a linked list of some sort for caching so that I could access it quickly. This would probably make more sense since the game engine itself is making the database query using interfaces to access the selected database server (right now I am just using MySQL for testing but it is set up to handle SQLite as well for single player games).
I don't know, maybe you are right about the bloat. Hmm. I will have to implement another idea bypassing any XML building/parsing and see if I can benchmark it. I am wide open to other ideas though, because I still don't have the logic in place to actually cache the data. The only thing I have in place is the XMLite parser which works very well so far. I think I will keep that in there for a small XML parser in case we decide to use it for the chat system (communication between the game server and chat server, or between the chat server and client).
This is why I wanted to post something on the forums. The GG community has always been good about jumping in and stopping the foolishness before it costs someone an arm or a leg. ;)
#3
03/09/2009 (5:49 am)
You know what? I think I am going to drop this idea altogether. Somehow it seemed like a workable idea at the time, but in the end, it will probably bloat our game. Besides, if we are using a server based database, then it will most likely be MySQL, and MySQL has some built in caching. I will just make sure that the MySQL server is sitting next to the game server so there is no delay in the network. Thanks for your input, it helped me clear my head. 
#4
do you have a notion how much data you might be talking about ?
i believe there are some open-source general purpose caching systems for MySQL, and as you mention MySQL may have some support for it itself. One approach might be to just ignore caching altogether until your game is at a point where it's actually a performance bottleneck. database caching only became a requirement for vSide when we started having about 800 simultaneous users, but of course that will vary wildly with the number and cost of the queries you're doing. (mind you those users would be spread across many torque servers which all connect to a single DB server)
as far as a dataformat for communicating between the database server and the torque engine, XML might be fine, i'm really not sure. the approach we took, which also introduces lots of bloat, was to urlencode key/value pairs, where sometimes a single value was itself a urlencoded list of key/value pairs. XML or some other encoding would probably be better, in fact.
hmm, somewhere on GG there's a resourced link to a recent article by a guy at either Intel or IBM describing a back-end architecture for an MMO based on torque, but i wasn't able to find it in a few minutes of searching. i'll try searching my gmail. .. huh, that's odd, it shows right up in gmail and the resource in question actually seems to be missing from the gg site. fortunately the article is still up at ibm:
Building a simple yet powerful MMO game architecture, Part 1: Introduction
Building a simple yet powerful MMO game architecture, Part 2: Gaming and Web integration
it's not precisely on-topic, but it's a short and good read anyhow, if you haven't already.
03/09/2009 (8:11 am)
heya - yeah totally, the GG community can be a great place just to bounce ideas around and think about them from other angles.do you have a notion how much data you might be talking about ?
i believe there are some open-source general purpose caching systems for MySQL, and as you mention MySQL may have some support for it itself. One approach might be to just ignore caching altogether until your game is at a point where it's actually a performance bottleneck. database caching only became a requirement for vSide when we started having about 800 simultaneous users, but of course that will vary wildly with the number and cost of the queries you're doing. (mind you those users would be spread across many torque servers which all connect to a single DB server)
as far as a dataformat for communicating between the database server and the torque engine, XML might be fine, i'm really not sure. the approach we took, which also introduces lots of bloat, was to urlencode key/value pairs, where sometimes a single value was itself a urlencoded list of key/value pairs. XML or some other encoding would probably be better, in fact.
hmm, somewhere on GG there's a resourced link to a recent article by a guy at either Intel or IBM describing a back-end architecture for an MMO based on torque, but i wasn't able to find it in a few minutes of searching. i'll try searching my gmail. .. huh, that's odd, it shows right up in gmail and the resource in question actually seems to be missing from the gg site. fortunately the article is still up at ibm:
Building a simple yet powerful MMO game architecture, Part 1: Introduction
Building a simple yet powerful MMO game architecture, Part 2: Gaming and Web integration
it's not precisely on-topic, but it's a short and good read anyhow, if you haven't already.
#5
Anyway, I don't want to bore you with details, but you can see there will be a LOT of database access going on for those elements that are dynamic. We had planned on defining the bare basics in the datablocks and then storing all additional information about an object in the database. The dialog tree will also be stored in a database. Storing this information in a database will also make it easier to create tools that we can use to update and modify game information.
Wow. I never pegged IBM as one to get involved in game software architecture. Thanks!
03/09/2009 (9:55 am)
Well, we are looking at probably as many gamers (or more) that play Planetside (still going and still fun). Our game idea is incorporating ideas from Eve Online, Planetside, and Earth and Beyond. We plan on having many different ship chassis and many different configurations for those ships. For example, one ship chassis can support 5 different configuration. Kind of like the old combat flight simulators, where you could choose your payload, except in ours, you can choose your payload as well as your electronic equipment, armor, and shields. Each weapon can be modified to enhance it (or debuff it if it was a negative modifier, like a virus attack). There will be multiple star systems, multiple planets (there is no plan to let the player actualy land on the planets, they are just something to look at), multiple space stations and multiple stationary capital ships. The stations themselves will be fully interactive, because the players will socialize in them, as well as attack and defend them. The stationary battleships will be smaller versions of the stations where players can gain a nearby spawn point to support a station attack (they have to take control of the cap ships in order to gain its benefits). Anyway, I don't want to bore you with details, but you can see there will be a LOT of database access going on for those elements that are dynamic. We had planned on defining the bare basics in the datablocks and then storing all additional information about an object in the database. The dialog tree will also be stored in a database. Storing this information in a database will also make it easier to create tools that we can use to update and modify game information.
Wow. I never pegged IBM as one to get involved in game software architecture. Thanks!
#6
03/09/2009 (4:47 pm)
Wow, this could actually be a good addition to Torque! You don't see many well thought out community resources lately. Yet this could save scripters and coders some stress when it comes to speed.
#7
One of the most critical points I will have to consider is which tables will use MyISAM and which tables will use InnoDB. Unfortunately, MyISAM has no support for foreign keys, which means the relational database design is a little more complicated (not much, just a little). However, MyISAM is a lot faster than InnoDB and when we are considering an MMO where there could be thousands of reads and inserts going on all the time, I would lean toward MyISAM. That does limit the flexability with locking though, so obviously not ALL the tables will be MyISAM. My brain is a little more transaction based, so I will have to research the design and figure out which tables will be handled through a transaction-like process. Those will need to be InnoDB. Tables that I am just spamming information to, such as logs, chat, etc., will probably be MyISAM. This is going to be quite a project. Lol!
03/10/2009 (9:07 am)
I am re-reading a MySQL book I bought when MySQL 5 came out titled "Database Design and Tuning" by MySQL Press. They talk about how they added the FEDERATED engine in version 5.0.3. Anyone ever used it? (maybe that is the built in caching tech I was hearing about) It sounds like it might be what I am looking for: "This feature defines table structures and operates upon data in remote locations as if they were local". The big problem I forsee is synchronization. Chances are very good that the game universe we are creating will be divided up on more than one server. This means, multiple servers updating and querying data. I still have to finish the IBM articles, and maybe they present a solution for data synchronization when we are using something like the FEDERATED engine, or some other caching technology. I still have a lot to read in many different resources.One of the most critical points I will have to consider is which tables will use MyISAM and which tables will use InnoDB. Unfortunately, MyISAM has no support for foreign keys, which means the relational database design is a little more complicated (not much, just a little). However, MyISAM is a lot faster than InnoDB and when we are considering an MMO where there could be thousands of reads and inserts going on all the time, I would lean toward MyISAM. That does limit the flexability with locking though, so obviously not ALL the tables will be MyISAM. My brain is a little more transaction based, so I will have to research the design and figure out which tables will be handled through a transaction-like process. Those will need to be InnoDB. Tables that I am just spamming information to, such as logs, chat, etc., will probably be MyISAM. This is going to be quite a project. Lol!
Associate Orion Elenzil
Real Life Plus
personally i'm a bit leery of XML as i think it can tend to be treated as a silver bullet and consequently lead to bad design, but it certainly can be used to great benefit. the places where i've seen it deliver the most value is in communication between disparate systems, for example one such use is described by ben garney in this blog post - he uses google spreadsheets to tweak game parameters and then has the game connect to the resulting XML feed. a very nice shared, 'agile' tool.
but using xml as a low-level data structure analagous to a struct makes me worry about memory and performance bloat. but i know many fine programmers who would say it's fine, so it's sort of a matter of opinion.
fwiw, we integrated TinyXML into vSide with a basic reflection in torque-script and it was pretty painless and quite effective for the task at hand, which was parsing some simple XML header data from media streams.