Storing Persistent Data in your Game
by Kneekick · in Torque Game Builder · 03/21/2006 (10:13 am) · 7 replies
This is my first game. I come from a background in Desktop & Web applications so I'm used to having SQL Server, MySQL, and even sometimes Access as my backend datasource when writing applications.
In game development, what options are there for persistent data storage that are able to be easily distributed with your game? I'm making a sports game so I will need to store team data, player data, stats, user preferences and some of this data will need to be able to be duplicated easily, for instance a user will have the ability to create multiple leagues in the game each league with it's own team data, division stats, player stats, etc...
Obviously I could use textfiles to store the data but I just wondering how other more experienced game developers choose to store their data. I hate to think parsing a delimited textfile is the best way of storing data. What is the consensus on what methods are faster, more reliable and easier to work with? And also what datasource support does TGB 1.1 have in it? I haven't seen much file or datasource operations in the tutorials, granted I've only gone through the basic and platformer and then peeked at the others.
Thanks.
In game development, what options are there for persistent data storage that are able to be easily distributed with your game? I'm making a sports game so I will need to store team data, player data, stats, user preferences and some of this data will need to be able to be duplicated easily, for instance a user will have the ability to create multiple leagues in the game each league with it's own team data, division stats, player stats, etc...
Obviously I could use textfiles to store the data but I just wondering how other more experienced game developers choose to store their data. I hate to think parsing a delimited textfile is the best way of storing data. What is the consensus on what methods are faster, more reliable and easier to work with? And also what datasource support does TGB 1.1 have in it? I haven't seen much file or datasource operations in the tutorials, granted I've only gone through the basic and platformer and then peeked at the others.
Thanks.
About the author
#3
Flat files can work great in some situations using the simple word/field parsing command already bulit into Torque. Then there's options like objects-to-xml serialization (there are resources for the engine for this out there), and as you've found MySQL and SQLite options too if you need that much power. I personally have used most of these options at one point or another. TorqueDB is another fantastic resource, offering transparent data serialization with some handy relational-like access methods.
I certainly wouldn't discount any of those out of hand, they're all very applicable in the right situations. I'd personally just recommend aim for what'll do and not be overkill. The immediate draw of a full DB implementation can be a bigger long-term hassle when you don't absolutely *need* the power.
03/23/2006 (9:34 am)
A lot of it depends on the specific case; how much data you're looking at, how you need to access it, is it primarily read-only or will there be constant updates, does it need to be secure, etc.Flat files can work great in some situations using the simple word/field parsing command already bulit into Torque. Then there's options like objects-to-xml serialization (there are resources for the engine for this out there), and as you've found MySQL and SQLite options too if you need that much power. I personally have used most of these options at one point or another. TorqueDB is another fantastic resource, offering transparent data serialization with some handy relational-like access methods.
I certainly wouldn't discount any of those out of hand, they're all very applicable in the right situations. I'd personally just recommend aim for what'll do and not be overkill. The immediate draw of a full DB implementation can be a bigger long-term hassle when you don't absolutely *need* the power.
#4
and for those that say well you have the source code you can add it yourself, I am here as a scripter not a c++.
I understand gg don't want to do it because they don't want to tie the code base to one db but they could create an addon product like the content packs. Think of all those rpg and stats based games that could be done.
03/24/2006 (3:33 am)
I have said this before it would be useful if they built sqlite in to t2d.and for those that say well you have the source code you can add it yourself, I am here as a scripter not a c++.
I understand gg don't want to do it because they don't want to tie the code base to one db but they could create an addon product like the content packs. Think of all those rpg and stats based games that could be done.
#5
03/24/2006 (5:22 am)
Building any form of SQL into a games code base by default is a very bad idea, it's so specialised it's only useful for one genre really, MMOGs and not everyone is making an MMOG.
#6
If you are making an mmog your gonna need some massive clustered databases and intregrated sql support just wouldn't cut it you would need to write the engine to work with what ever solution was choosen.
having it as an addon would work fine because it could be added only if the person requires it.
03/24/2006 (6:00 am)
I disagree tbh. sports titles with stats, rpgs, games with massive plots that require lots of storage. If you are making an mmog your gonna need some massive clustered databases and intregrated sql support just wouldn't cut it you would need to write the engine to work with what ever solution was choosen.
having it as an addon would work fine because it could be added only if the person requires it.
#7
If you are making a MMOG, then a SQL database seems justified. But, if you're making a small arcade style shareware title SQL just seems over the top.
Personally, I've pretty much started to like the old fashioned text files even more and more. In my experience, binary data storages can be a pain in the ass because every time you need to do something to the data, you'll have to create an editor or some specific utility for that. With text files, you can just open it with notepad or write a little piece of script to handle the thing. Only when the load/save time of some data is crucial, would I prefer binary over plain text.
Also, with proper text parser, writing and reading text files is just as quickly coded as writing/reading some binary file. (And with the text parser, I don't mean that it has to be some overbloated xml parser. ;) Some simple line/comma delimeter parser may do just as well.
Also, if you for some reason really want to write binary data, it may be worth a while to actually write that binary data in such format that it looks like its text... For example, use linefeeds/commas to seperate data entries and use ascii hex number notation (like 0xFFFF) instead of actual binary byte values (those hex values are still fixed-length like the binary data value would be, just a bit longer). You can still read in/write the entries as binary, without having to actually parse those linefeeds and other seperators, they are just acting as fillings here and there. This way you can actually open the file in notepad to view it, and you can even modify the data as long as you don't add/delete characters. Also, if you later on would like to change to text format / text parser, your data will already be text "compatible". You'll only lose a bit in data storage size, but gain in maintenance/debugging ease and future extendability.
If you don't want the text data to be readable by others, then just crypt the text or create some kind of compiled format of the data (see torque's .cs -> .cs.dso compilation).
And if you are really worried about text data being so damn big compared to binary, just zip it and you won't see much difference.
Oh, and if you are going to have a scripting language in your program, then most definately the preferred approach for a lot of data is to actually save it in script format. (see torque's mission and gui files).
The point is saving in script format is simple... If you are going to have scripting in your game, you'll need some script commands to modify these data values anyway. So why create 2 different formats to express the same data, when you can just use one. This way you can focus on creating a single robust text parser for the scripts and then re-use that elsewhere for free, instead of creating two crappy text parsers, one for data and one for scripts (and then ending up making the same data modification commands to the scripting language that you just made to the data reader).
In overall, I think that torque very well demonstrates this to be a well working approach. (One of the reasons I like torque ;).
Of course, if you are talking about some volatile data that keeps changing constantly at random locations, the parsed plain text format is bad. Then I'd go with the text-format compatible binary format (as described above). An example of badly designed plaintext format is the standard unix mailbox. In worst case, when you want to flag a single mail read (in other words, add a single character), you may have to rewrite the whole mailbox file (a perfect way to kill a mail server). So, any sensible mail reader agent will probably either use some other mail file format or will extend the unix mailbox format. (Although I've seen some less sensible mail agents too).
I've pretty much learned these lessons myself while working with web portal applications as well as games.
To sum it up: Random binary data is bad, text parser compatible binary is ok, plain text in standardized format is good, script format is great. ;)
I'm not quite sure yet how easy it might be to create some minimal database like functionality by writing large amounts of data in torquescript storage... That might be an interesting task to do, if I had the time for it. :P
03/25/2006 (1:31 pm)
I'd say that this depends totally on the game... If you are making a MMOG, then a SQL database seems justified. But, if you're making a small arcade style shareware title SQL just seems over the top.
Personally, I've pretty much started to like the old fashioned text files even more and more. In my experience, binary data storages can be a pain in the ass because every time you need to do something to the data, you'll have to create an editor or some specific utility for that. With text files, you can just open it with notepad or write a little piece of script to handle the thing. Only when the load/save time of some data is crucial, would I prefer binary over plain text.
Also, with proper text parser, writing and reading text files is just as quickly coded as writing/reading some binary file. (And with the text parser, I don't mean that it has to be some overbloated xml parser. ;) Some simple line/comma delimeter parser may do just as well.
Also, if you for some reason really want to write binary data, it may be worth a while to actually write that binary data in such format that it looks like its text... For example, use linefeeds/commas to seperate data entries and use ascii hex number notation (like 0xFFFF) instead of actual binary byte values (those hex values are still fixed-length like the binary data value would be, just a bit longer). You can still read in/write the entries as binary, without having to actually parse those linefeeds and other seperators, they are just acting as fillings here and there. This way you can actually open the file in notepad to view it, and you can even modify the data as long as you don't add/delete characters. Also, if you later on would like to change to text format / text parser, your data will already be text "compatible". You'll only lose a bit in data storage size, but gain in maintenance/debugging ease and future extendability.
If you don't want the text data to be readable by others, then just crypt the text or create some kind of compiled format of the data (see torque's .cs -> .cs.dso compilation).
And if you are really worried about text data being so damn big compared to binary, just zip it and you won't see much difference.
Oh, and if you are going to have a scripting language in your program, then most definately the preferred approach for a lot of data is to actually save it in script format. (see torque's mission and gui files).
The point is saving in script format is simple... If you are going to have scripting in your game, you'll need some script commands to modify these data values anyway. So why create 2 different formats to express the same data, when you can just use one. This way you can focus on creating a single robust text parser for the scripts and then re-use that elsewhere for free, instead of creating two crappy text parsers, one for data and one for scripts (and then ending up making the same data modification commands to the scripting language that you just made to the data reader).
In overall, I think that torque very well demonstrates this to be a well working approach. (One of the reasons I like torque ;).
Of course, if you are talking about some volatile data that keeps changing constantly at random locations, the parsed plain text format is bad. Then I'd go with the text-format compatible binary format (as described above). An example of badly designed plaintext format is the standard unix mailbox. In worst case, when you want to flag a single mail read (in other words, add a single character), you may have to rewrite the whole mailbox file (a perfect way to kill a mail server). So, any sensible mail reader agent will probably either use some other mail file format or will extend the unix mailbox format. (Although I've seen some less sensible mail agents too).
I've pretty much learned these lessons myself while working with web portal applications as well as games.
To sum it up: Random binary data is bad, text parser compatible binary is ok, plain text in standardized format is good, script format is great. ;)
I'm not quite sure yet how easy it might be to create some minimal database like functionality by writing large amounts of data in torquescript storage... That might be an interesting task to do, if I had the time for it. :P
Torque Owner Kneekick
I'm would like to know what methods other more experienced game developers than myself are using in their torque games and what maybe they would recommend I use in mine. I need something that easily portable that the user does not have to setup and does not need to connect over the internet to a database server. Something local with a small footprint. Right now SQLite is the front runner in my mind. But that's only from what I've read and my limited knowledge of the options available to me.
Just trying to pick other developers brains and see what falls out. :)