Game Patcher
by John Spivey · in General Discussion · 02/09/2006 (7:41 am) · 29 replies
Today I have been looking for 3rd party software providers that have a simple "patcher" as we have all come to love in those MMO's we play so much. It needs to check to see if the end user has the same files as the client. So if i accidentally corrupt my .mis file all i need do is delete it. The patcher will see that its gone then replace it from the patch server.
I thought this would be an easy thing to find but as it turns out most patching software talks more about network security...blah blah blah. So I am not sure if these aplications would work for what I am trying to do here.
Simply put out team is making an MMO and we need to get a patcher program in there and running. Does anyone here have any idea where I might find an application like this?
Thanks for any help you can give.
I thought this would be an easy thing to find but as it turns out most patching software talks more about network security...blah blah blah. So I am not sure if these aplications would work for what I am trying to do here.
Simply put out team is making an MMO and we need to get a patcher program in there and running. Does anyone here have any idea where I might find an application like this?
Thanks for any help you can give.
#22
1. Anyone using rdist as a patcher should be very careful how they use it - specifically, allowing rdist to delete files not on the server can be very dangerous. Consider the possibility of a (clueless) user installing your game in c:\windows - then your patcher comes along and deletes all those "useless" files in the same directory as your game! Ideally, any patcher that deletes files should only be allowed to delete very specific files that you specify as obsolete. In fact, it's generally a good idea to be very careful whenever you delete files, including when you uninstall your game and any times you delete files inside your game.
2. Using any binary patcher on a large zip file containing all your game data is probably going to create much larger patch files than you'd expect. Even if only one file in the zip has changed, your patch file (or partial rsync download) might still be nearly as large as the original zip. This is because the patcher will compare the two files starting at the beginning until it finds the first difference, and then it will update from there. If the modified file in your zip is early in the file (or if the header is different), the patcher will include everything from that point on in the patch. A very smart patcher might not do this (it looks like bsdiff may not), but I think the partial update feature of rsync would. One way around this would be to allow your patcher to download the actual files that have changed and then freshen the zip with the new files.
03/13/2006 (11:01 am)
I was revisiting this thread, and a few points came to mind:1. Anyone using rdist as a patcher should be very careful how they use it - specifically, allowing rdist to delete files not on the server can be very dangerous. Consider the possibility of a (clueless) user installing your game in c:\windows - then your patcher comes along and deletes all those "useless" files in the same directory as your game! Ideally, any patcher that deletes files should only be allowed to delete very specific files that you specify as obsolete. In fact, it's generally a good idea to be very careful whenever you delete files, including when you uninstall your game and any times you delete files inside your game.
2. Using any binary patcher on a large zip file containing all your game data is probably going to create much larger patch files than you'd expect. Even if only one file in the zip has changed, your patch file (or partial rsync download) might still be nearly as large as the original zip. This is because the patcher will compare the two files starting at the beginning until it finds the first difference, and then it will update from there. If the modified file in your zip is early in the file (or if the header is different), the patcher will include everything from that point on in the patch. A very smart patcher might not do this (it looks like bsdiff may not), but I think the partial update feature of rsync would. One way around this would be to allow your patcher to download the actual files that have changed and then freshen the zip with the new files.
#23
03/13/2006 (11:38 am)
I had noticed that if you connect to a server m(the more updated version of your game) That the engine would start downloading anything that you are missiong. All but the .mis file. Thoe, i dont think it looks for things that have changed, just the ones that are not there. 
#24
@Allyn: That's relevant for your TGE/TSEers, but us T2Ders don't have that sort of luxury. :) For TGE/TSE I would probably choose to have my own front-end in front of the binary that did the version check and patching before actually running the game -- that way you can do things like download a patch for the game binary.
-JF
03/13/2006 (12:50 pm)
@Philip: Your second point is actually a very good one. Even bsdiff will not be able to avoid downloading everything afterwards: It's just the nature of compressed/encrypted data that the data will all be different after the changepoint (or depending on the compression algorithm, even from the beginning of the file...). If you're going to do incremental updates and you intend to use compression or encryption, breaking your resources up into smaller .zip files -- ideally grouped by likelihood of changing together -- is an important step in ensuring that you can incrementally download changes.@Allyn: That's relevant for your TGE/TSEers, but us T2Ders don't have that sort of luxury. :) For TGE/TSE I would probably choose to have my own front-end in front of the binary that did the version check and patching before actually running the game -- that way you can do things like download a patch for the game binary.
-JF
#25
03/17/2006 (9:47 am)
I was under the impression that Rsync only sent the differences, but upon trying it.. it sent the whole file.
#26
03/17/2006 (10:27 am)
@Stefan: Were you trying to sync a compressed file (.zip file)? It's basically impossible to send "onlt the differences" of a compressed file. For point of reference, where I work, we have several large files (.war files) that need to get synchronized into our production environment when we do a release. If we made the .war file compressed (50-70MB) it transferred basically the entire file. If we made the .war file UNcompressed (100-120MB) it sent only the changes -- MUCH MUCH faster.
#27
Is this because, when adding a file to a zip, the whole structure is changed, and thus the delta is so much different that the full file will be sent?
In any case, thanks for the info Jon.
03/20/2006 (5:19 pm)
I suspected that, but never tried without compression.Is this because, when adding a file to a zip, the whole structure is changed, and thus the delta is so much different that the full file will be sent?
In any case, thanks for the info Jon.
#28
you may have changed only a few bytes of the original file,
but an arbitrary number of bits in the archive may have changed.
03/20/2006 (5:36 pm)
Stefan - i believe that's the case in general w/ huffman-style encoding.you may have changed only a few bytes of the original file,
but an arbitrary number of bits in the archive may have changed.
#29
An example to explain this: In (Adaptive) Huffman coding, the compressor might decide to represent the letter "A" with the bit "1", the letter "B" with the bits "01", and the letter C with the bits "001". So the text:
AACBBAABAB becomes "1100 1010 1110 1101" (2 bytes)
But let's say you change it to:
AABBBAABAB
The compressor might instead make "A" be "1", and "B" simply be "0". So your message becomes:
"1100 0110 10[00 0000]" (zero-padded to make even bytes)
The first four bits remain the same, but everything after that is altered. This is incredibly over-simplified, but that's the basic idea. Dictionary coding (LZ[W] and variants) can become similarly jumbled for similar reasons.
-JF
03/20/2006 (10:39 pm)
The "deflate" algorithm usually employed when creating .zip files is a combination of the LZ77 algorithm and Huffman coding. The former is a dictionary-based approach, the latter is a tree-based approach. Either approach is susceptible to having the whole structure of the file change due to a small change in the data.An example to explain this: In (Adaptive) Huffman coding, the compressor might decide to represent the letter "A" with the bit "1", the letter "B" with the bits "01", and the letter C with the bits "001". So the text:
AACBBAABAB becomes "1100 1010 1110 1101" (2 bytes)
But let's say you change it to:
AABBBAABAB
The compressor might instead make "A" be "1", and "B" simply be "0". So your message becomes:
"1100 0110 10[00 0000]" (zero-padded to make even bytes)
The first four bits remain the same, but everything after that is altered. This is incredibly over-simplified, but that's the basic idea. Dictionary coding (LZ[W] and variants) can become similarly jumbled for similar reasons.
-JF
Torque Owner Jon Frisby
MrJoy, Inc.
That approach definitely has some advantages -- less computational overhead for the client and the server being the most notable. The drawback is you have to produce patchsets from every version to every subsequent version and then make the client download the right one. It's not a huge problem, but it's more legwork... I think this qualified as a classic "time/space tradeoff" optimization though. :)
-JF