Disaster Recovery - for the Enterprise Indie

A recent .plan from Johnathan increased my interest in the indie prep for a disaster. Johnathan's script is an excellent starting point. After all, we are the smallest of small businesses and most of those have the barest of plans in place. From here on I'll refer to Disaster Recovery simply as DR. This is a vast topic even to the bounds that affect us.

Don't eject just yet. Indies don't have the luxury of outsourcing DR. We have to be the ones to carve out its plans to the degree that our projects require. Some of you are more ready for this than others. This .plan covers what I've put in place and will hopefully elicit power up feedback. There is a point where the levels of planning become impractical but prefer to err on the side of caution. The Perl script that I use for the backup portion is available for download at the end of this post.

The Basics:
1. Organize data and workflow. NDA (non-disclosure agreement) anyone who has access to either. Define its storage, retrieval, updating, and which tools are used for each. Don't assume that everyone knows what to use or you'll get some guy writing documentation in Gimp XCF files. Not that there's anything wrong with that; it's just not the most collaborative (using multiple layers per text entry isn't revision control).

At the file system level, use one working directory per game project that includes all its related art, code, scripts, design docs, scanned copies of NDAs and all other legal docs, and engine subdirs. That's right multiple engine subdirs.

There should be at least two engine related directories; the first one being a customized Torque for the dream game in production and the second is a vanilla, stock TGE install for comparisons when things get real crazy. Never make any changes to the vanilla TGE. It could make the moon split in half or Linus to intern at Microsoft or something else equally sinister.

2. Clearly define job roles for both before and after a disaster (if you have the luxury of a team). More than likely the survivors will be wearing more hats than usual, at least for the short-term. Don't go crazy, but take reasonable disasters into consideration; fire, flood, earthquake, volcanoes, cats and dogs living together, whatever are common to your locale. How long will a programmer need to hack together placeholder models until the artist gets back online? Make sure that everyone has adequate contact information and knows what to do during the first 3-days to one-week of a disaster.

3. Everybody documents; no exceptions. It helps to have pre-defined templates with boilerplate to get the writing unblocked. Anything above-and-beyond the boilerplate is gravy.

4. Define a check out duration. Define limits on the amount of time an asset can be away from the protective covering of the backup policy. For example, an artist can work on a model at a remote location for 8 hours; after which it must be placed under the directories being backed up. Version control software helps to keep these checks in place. Some large companies I've worked for use end-of-day reports to verify if any assets are outside version control. We have far less do over leeway than the big boys.

5. Now that all game content is in a fixed location, set backup policies...and test them regularly.

6. Move the backup media off-site on a regular basis. Preferably as far away from your disaster zone as possible. If you live in a flood plane make sure that media is outside its bounds.

Everyone technical, or artistic, enough to work on a game project knows in their heart of hearts that backups are crucial; but that's only part of the picture. The extra media does nobody any good if it is within the disaster.

Here's my Perl script that zips up all subdirectories into their own individual files within the backup directory. The script can be downloaded here. Zip files are stamped with the hour and minute so that multiple backups can be taken throughout the day if needed without overwriting the previous. Alternatively, the code also allows the files to be prefixed with the weekday name. There are three possible times during the day when I might be able to work on my game. This script is scheduled to run after those times so as to catch any changes.

It's GPL so make it fit your project as needed. I haven't tested on Linux or Mac yet so beware. By default it processes all subdirectories from the current working directory. There are imported modules used too. I tried to comment heavily and steer clear from using the more clumbersome Perl syntax.

The backup directory is burned to DVD-RW and rotated away from my home and office. It's impossible to consider SFTP as a reliable transport to a distant web server. External hard drives are a nice convenience but are only good in packs as they must be rotated off-site. For now, all my backups fit onto a DVD which has a acceptable quick read/write time.