Minidump Support in TGEA

Download Code File

Target Audience
This resource is geared towards teams looking to improve the stability of their TGEA product releases. The modifications implement mindump support on the windows platform, but do not violate Torque's platform layer (so support can be extended to the XBox or other platforms and platforms with no minidump support will not be effected). Non-programmers and sole developers will likely find little use for this resource. Also, if you don't know how to modify your project to include files and to change compilation options, this probably isn't for you.

This resource adds minidump support to TGEA for project configurations that define TORQUE_RELEASE. So, minidump support is not included in debug builds or final builds. In addition to creating the minidump (in a location specified in script), the code will also copy the console.log, executable, and pdb files to the crash folder, and will create a crash log that includes comments from the user and a stack trace. Finally, the code is configurable via script to allow for automated email notifications or some other user specified action.

What the heck is a minidump?
Briefly, minidumps are used to track down crash bugs that occur in release versions of software. When a crash occurs a minidump file is created to save the call stack, state of registers, etc. at the time of the crash. Programmers can then use that information to track down the cause of the crash. In my experience, having minidump support coupled with automated notifications, greatly improves a products stability. For a more thorough explanation of minidumps, please read this article from code project.

Code Modification Steps
1) Backup your work.

2) Copy the following files over to the winPlatform folder and add them to your project:
winStackWalker.h
winStackWalker.cpp
winMiniDump.cpp

3) Copy the following files over to the platform folder and add them to your project:
platformStackWalker.h

4) Merge the following files in your project. (If you are using stock TGEA 1.01, then you can simply replace your files with those included.) All of the changes are indicated by //TG-BEGIN and //TG-END comment blocks.
platform.h
winStrings.cpp
winWindow.cpp
main.cpp
fileio.h

Thats it. At this point, whenever a crash occurs with a TGEA build that defines TORQUE_RELEASE, a minidump will be created. By default, crash directories are created within a directory called MiniDump. Within the minidump directory, every time a crash occurs a new directory is created using the name of the executable, the date, and the time of the crash. Within the individual crash directories the minidump file, the console log, the executable, the pdb, and a crash log (with user input and a stack trace) will be saved. Because the individual crash folder are time stamped, it is easy to find a specific crash.

Using the MiniDumps
In order to provide programmers with the most information, you should set your compiler options to generate debug information (.pdb files) in your release builds. (Note that I am assuming you have debug, release, and final configurations.) If you output the .pdb files into the same directory as your executables and name the .pdb files the same as your executable, then the crash system will always have the proper .pdb for a given crash. You do not need to change any of the optimizations, so your release build will still be fast. For example, to change the stock TGEA project you would go to properties->Linker->Debugging and set "Generate Debug Info" to Yes. You would then "Output Program Database File" to be "../example/TGEA.pdb" so that the output location is the same as the executable.

The .dmp files can be opened in visual studio. You can then right click on the dmp in visual studio and select 'debug->start new instance'. This will show you where the crash occurred along with the call stack. However, variable values are not to be trusted, they are often wrong. Register values, on the other hand, are usually correct (so detecting a NULL pointer, for example, is fairly straight forward).

Extending Functionality via Script
To really get the full benefit of using crash dumps, you will want to customize the behavior in script. Specifically, there is a small set of script variables that can be used to automate notifications, customize save locations, etc. I set this variables in the common/main.cs function InitCommon() so that they would be set for all mods, but that is only a suggestion.

$MiniDump::dir = "k:/MiniDump";
This variable can be used to specify the location of the Minidump directory that will contain all of the crash directories. By specifying a directory to which all team members have access (for example a shared LAN drive), a designer could crash the program and a programmer would be able to access the minidump information. In the above example, crashes would be saved to k:/MiniDump.

$MiniDump::exec = "blat.exe"
Once the minidump information has been saved to the specified folder, the code will attempt to execute the program specified by the $MiniDump::exec script variable. This can be used to fire off an email alert or any other custom functionality. In the above example I using blat.exe to send an email notification. (In actuality, we use a custom python script that I will make available once I have cleaned it up. Because people are working remotely, the python script is used to zip up the files and includes them in an email. It also allows for username/password logins to remote email.)

$MiniDump::params = "\"%crashlog%\" -subject %crashfolder% -to crashReport@tormentedgames.com -server smtp.bizmail.yahoo.com -f todd.scott@tormentedgames.com"
Specifying blat as the executable wouldn't be of much use if we couldn't specify the parameters. So, the $MiniDump::params variable allows you to do just that. Additionally, the following indicators can be specified in the parameter string to customize the functionality on a per crash basis:

%crashlog% will be replaced with a fully specified path to the crash log
%crashfolder% will be replaced with the name of the specific crash folder
%crashpath% will be replaced with the fully specified path of the crash folder

In the above example, an email will be sent from todd.scott@tormentedgames.com to crashReport@tormentedGames.com. The subject line will be the time stamped folder that contains the crash (making it very easy to find whenever a programmer gets around to it). The contents of the email will contain any user comments entered into a dialog box at the time of the crash and a stack trace showing the state of the call stack when the crash occured (very helpful for identifying similiar or known crashes). Instead of hard coding who the email is from, you could use a script variable that would pull the current user from an environment variable (We use the python script that I mentioned to have users provide a username and password so the crash log can be sent).

Future Functionality
Overall, I'm pretty happy with the functionality. However, I would like to improve the stack trace to give more information (like function parameter names). I would also like to add word wrapping to the windows dialog user text section and I would like to make it so that the user cannot move the dialog box around. (However, windows dialog boxes aren't my expertise, so if someone else already knows how to do this or were to look into it, I would be greatful.)

Additionally, if there is functionality that people would like to see added, please let me know. I am happy to extend the resource to make it more useable. Please report any errors or problems that you have, and I will fix them as soon as possible.