Deploying Indie Games
by Konrad Kiss · 06/09/2011 (6:33 am) · 27 comments
Whether it's a game or another kind of desktop application, deploying it to your users can be a pain if you can't afford to or don't want to pay the price for the high-end tools. There are alternatives, however. And there are pitfalls.
This has been posted on my personal blog but I thought it would hopefully benefit the community if I re-posted it in my GG blog. Hope it saves you the headaches I had.
Since the game my studio works on is an MMO, we had to keep all of the game logic strictly on the server side. This means that the game server is authoritative and the client is merely a "screen" to display what the server is computing.
Still, protecting game assets in the client version of Xenocell is a very important task. Not so much because people can steal our stuff, but also because we use assets that assume such protection and we agreed to abide by their license agreements - such as an adequate level of protection of these assets against 3rd party use.
So after taking a good look around, we decided to go with Molebox VS to pack our game's contents. This solution has the benefit of being transparent to the game code. It functions as a virtual file system so we did not have to make any changes to the existing code, which was a huge benefit.
Alas, as I had to learn, virus and malware authors also found this packing solution elegant. Although Teggo Software - the studio behind Molebox - ensures its customers that it has a "unique executable template for every customer. This helps preventing antivirus false positives since applications of different users look less similar." - it doesn't actually work as advertised, unfortunately. Interestingly enough, I don't think it's Teggo's fault.

Most users can't tell the difference between a false positive and malware.
You see, anti-virus software needs to be fast to stay ahead of the competition. Almost all of these solutions default to marking your executable suspicious if it was packed with a packer that has been used before for creating malware. They can't always look behind the protections employed in the packer to actually know if your game executable is indeed dangerous or not, so they assume it is. There's simply no time for the AV to crack the protection and take a peek at your binary data.
It is very bad practice to assume that anything that has been packed with Molebox (or any other packer for that matter) is dangerous. But it's safe for the user, and I'm not sure I could come up with a better solution. I'm sure however that these alerts hurt many businesses that become aware of the issue only when it's too late.
I don't dare to assume that I could stop dedicated crackers from getting to what they want, and it would be a mistake to think I ever could - but if I can keep our assets safe from 99% of the users then it's a compromise I can live with. For an online game with a privately hosted authoritative server, the decision was easy: be as transparent to anti-virus software as possible. So I opted to not use Molebox's advanced encryption and obfuscation mechanisms, and - it worked! I did a few quick checks to see how the executable did and the results were great!
But this wasn't the end of the story. The executable still bears the marks of our packer, and during the installation process it again gets packed into the setup executable. I assumed that AV programs will be able to see inside the installer binaries, perhaps check the individual files, but they will - again - not bother checking them thoroughly. I was right, unfortunately, because AV software reported the setup .exe as a false positive right at the moment we tried to download it during our deployment tests.
For patching and installation we use Putchisoft's Dispatcher - it is a very powerful tool with binary difference patching capabilities for a very indie-friendly price. Not only that, but Robin, the guy behind Dispatcher offers one of the the best customer support I've ever seen, and he is also available for custom modifications to the software. (We worked together with him to find a way for Dispatcher to be able to handle patches published via Amazon's CloudFront service while avoiding its caching mechanism to make patches available around the globe instantly for everyone to download.)
So with Dispatcher, I managed to go another route. Since installation (NSIS) is a part of Dispatcher's deployment process, I changed our setup executable to not include any of the game files, but download them instead as a part of the install process. Dispatcher has a built-in way to automatically download the latest version if the installation is damaged beyond repair. Not having any of the game files in the installation directory pretty much satisfied this condition.
This approach had a number of advantages. First, what mattered most to me was that the game executable was being transmitted in a fashion that was again transparent to AV software - it could easily confirm that the game executable is not harmful. On the other hand, it was less than 200 kilobytes in size, which was a great side-effect. The third best thing was that running it would always get the latest version of our game. The only drawback was that it required an Internet connection at install time. Since our game assumes a constant Internet connection, this didn't bother me much.
I guess that's it. False positive alerts from anti-virus software could have hurt our business significantly, but in the end instead of more obfuscation, transparency proved to be the right way to go.
As always, thanks for reading - any comments are much appreciated (either here or on my blog). I hope to continue my AWS series that I started with my previous blog post with an example on how to host your game downloads in the cloud and how to make it available with mirrors all over the world.
@konradkiss
KonradKiss @ GitHub
konradkiss.com
This has been posted on my personal blog but I thought it would hopefully benefit the community if I re-posted it in my GG blog. Hope it saves you the headaches I had.
Since the game my studio works on is an MMO, we had to keep all of the game logic strictly on the server side. This means that the game server is authoritative and the client is merely a "screen" to display what the server is computing.
Still, protecting game assets in the client version of Xenocell is a very important task. Not so much because people can steal our stuff, but also because we use assets that assume such protection and we agreed to abide by their license agreements - such as an adequate level of protection of these assets against 3rd party use.
So after taking a good look around, we decided to go with Molebox VS to pack our game's contents. This solution has the benefit of being transparent to the game code. It functions as a virtual file system so we did not have to make any changes to the existing code, which was a huge benefit.
Alas, as I had to learn, virus and malware authors also found this packing solution elegant. Although Teggo Software - the studio behind Molebox - ensures its customers that it has a "unique executable template for every customer. This helps preventing antivirus false positives since applications of different users look less similar." - it doesn't actually work as advertised, unfortunately. Interestingly enough, I don't think it's Teggo's fault.

Most users can't tell the difference between a false positive and malware.
You see, anti-virus software needs to be fast to stay ahead of the competition. Almost all of these solutions default to marking your executable suspicious if it was packed with a packer that has been used before for creating malware. They can't always look behind the protections employed in the packer to actually know if your game executable is indeed dangerous or not, so they assume it is. There's simply no time for the AV to crack the protection and take a peek at your binary data.
It is very bad practice to assume that anything that has been packed with Molebox (or any other packer for that matter) is dangerous. But it's safe for the user, and I'm not sure I could come up with a better solution. I'm sure however that these alerts hurt many businesses that become aware of the issue only when it's too late.
I don't dare to assume that I could stop dedicated crackers from getting to what they want, and it would be a mistake to think I ever could - but if I can keep our assets safe from 99% of the users then it's a compromise I can live with. For an online game with a privately hosted authoritative server, the decision was easy: be as transparent to anti-virus software as possible. So I opted to not use Molebox's advanced encryption and obfuscation mechanisms, and - it worked! I did a few quick checks to see how the executable did and the results were great!
But this wasn't the end of the story. The executable still bears the marks of our packer, and during the installation process it again gets packed into the setup executable. I assumed that AV programs will be able to see inside the installer binaries, perhaps check the individual files, but they will - again - not bother checking them thoroughly. I was right, unfortunately, because AV software reported the setup .exe as a false positive right at the moment we tried to download it during our deployment tests.
For patching and installation we use Putchisoft's Dispatcher - it is a very powerful tool with binary difference patching capabilities for a very indie-friendly price. Not only that, but Robin, the guy behind Dispatcher offers one of the the best customer support I've ever seen, and he is also available for custom modifications to the software. (We worked together with him to find a way for Dispatcher to be able to handle patches published via Amazon's CloudFront service while avoiding its caching mechanism to make patches available around the globe instantly for everyone to download.)
So with Dispatcher, I managed to go another route. Since installation (NSIS) is a part of Dispatcher's deployment process, I changed our setup executable to not include any of the game files, but download them instead as a part of the install process. Dispatcher has a built-in way to automatically download the latest version if the installation is damaged beyond repair. Not having any of the game files in the installation directory pretty much satisfied this condition.
This approach had a number of advantages. First, what mattered most to me was that the game executable was being transmitted in a fashion that was again transparent to AV software - it could easily confirm that the game executable is not harmful. On the other hand, it was less than 200 kilobytes in size, which was a great side-effect. The third best thing was that running it would always get the latest version of our game. The only drawback was that it required an Internet connection at install time. Since our game assumes a constant Internet connection, this didn't bother me much.
I guess that's it. False positive alerts from anti-virus software could have hurt our business significantly, but in the end instead of more obfuscation, transparency proved to be the right way to go.
As always, thanks for reading - any comments are much appreciated (either here or on my blog). I hope to continue my AWS series that I started with my previous blog post with an example on how to host your game downloads in the cloud and how to make it available with mirrors all over the world.
@konradkiss
KonradKiss @ GitHub
konradkiss.com
About the author
http://about.me/konrad.kiss
#22
Can you make an estimate on approximately how long it too you to integrate it into your project? I'm late in the development stage, and have been considering an asset packaging system as well, considering strengths and weaknesses. We're strictly a single player game and everything will be on the client. I'd like to package up the assets but need to evaluate that against the additional time to integrate the package, as we're so close to the end.
Thanks!
Dusty Monk
(and good luck with your launch!)
06/13/2011 (6:24 am)
Great blog post Konrad, and some excellent suggestions. Quick question for you for Molebox:Can you make an estimate on approximately how long it too you to integrate it into your project? I'm late in the development stage, and have been considering an asset packaging system as well, considering strengths and weaknesses. We're strictly a single player game and everything will be on the client. I'd like to package up the assets but need to evaluate that against the additional time to integrate the package, as we're so close to the end.
Thanks!
Dusty Monk
(and good luck with your launch!)
#23
Molebox needs no integration into your code, it wraps around your exe. However, it will pose AV issues depending on your Molebox settings. The stronger security you employ, the more AV programs will alert the user for a possible malware. It's hard to say an estimate, because your time would mostly be spent doing trial and error checks on your packed executable.
I promote weak encryption settings for Molebox, but it isn't good enough for games where the user can alter online scores by being able to access code in the executable. So it would only work if the server is hosted privately and in an authoritative fashion, or if there is no multiplayer aspect (even as little as an online leaderboards feature) at all in the game. For everything else, where hacking the client poses a danger that goes beyond the actual client you probably need more security, and Molebox there will give you more headaches than help.
So if you have any communal feature in you game, then Molebox is not your best bet imho.
Please, if you ever find good alternatives, let us know.
It would be nice to have the ability to "cook" deployed versions right from Torque. The deployment process is entirely neglected. .dso compilation is simply not enough.
06/13/2011 (9:01 am)
@Dusty: Thank you!Molebox needs no integration into your code, it wraps around your exe. However, it will pose AV issues depending on your Molebox settings. The stronger security you employ, the more AV programs will alert the user for a possible malware. It's hard to say an estimate, because your time would mostly be spent doing trial and error checks on your packed executable.
I promote weak encryption settings for Molebox, but it isn't good enough for games where the user can alter online scores by being able to access code in the executable. So it would only work if the server is hosted privately and in an authoritative fashion, or if there is no multiplayer aspect (even as little as an online leaderboards feature) at all in the game. For everything else, where hacking the client poses a danger that goes beyond the actual client you probably need more security, and Molebox there will give you more headaches than help.
So if you have any communal feature in you game, then Molebox is not your best bet imho.
Please, if you ever find good alternatives, let us know.
It would be nice to have the ability to "cook" deployed versions right from Torque. The deployment process is entirely neglected. .dso compilation is simply not enough.
#24
I second that wish. Maybe something that uses the build-for-web feature in T3D as a starting point?
06/13/2011 (9:07 am)
@KonradI second that wish. Maybe something that uses the build-for-web feature in T3D as a starting point?
#25
Can I get some help with MoleBox.
Yesterday, I downloaded a demo version of it. I made one executable file including all related files. And when I hit the start button after loading splash gui, my game stopped there saying "waiting for server."
Have you ever noticed like this?
06/13/2011 (10:17 pm)
@Konrad,Can I get some help with MoleBox.
Yesterday, I downloaded a demo version of it. I made one executable file including all related files. And when I hit the start button after loading splash gui, my game stopped there saying "waiting for server."
Have you ever noticed like this?
#26
06/14/2011 (1:54 am)
@game4Rest: I have seen this before and if I remember correctly, it was due to me packaging files that must be written to at runtime. Try excluding stuff like prefs.cs, config.cs, procedural shaders, cached fonts, etc... Also, don't use dso. Exclude the debug versions of the binaries, and core/scripts/GUI/messageBoxes as weird as it might sound, that directory seems to be causing MB some trouble that I did not investigate. You don't need the tools and web folders either. One of these should fix that problem. :) Good luck, let me know how it went! 
Torque Owner TheGasMan
G.A.S. [+others]
[note to anyone looking into that patcher: If you sign up for TrialPay & buy a $5 game from BigFish, You should get a license to the pro version if completed correctly. (I'm not a trial pay employee etc... but to explain;
I purchased a Winrar license for a client the other day, through trial pay I quickly picked a $5 game:world of goo, from Bigfish games, I received the Winrar license file minutes later.
..So that saved them about 25$ & I'm not sure if that deal goes outside the US. ]
..but $5 for an updater; sweet.