Zip code in ResourceManager
by James Urquhart · in Torque Game Engine · 09/09/2004 (2:27 pm) · 5 replies
Hi there,
I'm a bit puzzled by the way files in .zip's are handled by the ResourceManager in torque.
Specifically, these functions :
void ResManager::scanZip (ResourceObject * zipObject)
void ResManager::searchPath (const char *path)
bool ResManager::setModZip(const char* path)
void ResManager::setModPaths (U32 numPaths, const char **paths)
From what i can tell, all the paths start in setModPaths(). The function then calls both setModZip() and searchPath() on each path found.
The real strange thing is that both setModZip() and searchPath() (along with scanZip()) open a zip, and add files from it. But from what i can tell, they are basically doing the same exact thing, which seems to be quite a pointless duplication of effort.
Does anyone have any ideas why it seems to be doing this?
Currently i have rewritten my zip handling code to handle a custom container format, and i have only implemented the searchPath() route.
I've not got files loading from the container yet (resource manager seems to think they do not exist, though they apear in the resource list) - though i'm getting closer to a solution ;)
I'm a bit puzzled by the way files in .zip's are handled by the ResourceManager in torque.
Specifically, these functions :
void ResManager::scanZip (ResourceObject * zipObject)
void ResManager::searchPath (const char *path)
bool ResManager::setModZip(const char* path)
void ResManager::setModPaths (U32 numPaths, const char **paths)
From what i can tell, all the paths start in setModPaths(). The function then calls both setModZip() and searchPath() on each path found.
The real strange thing is that both setModZip() and searchPath() (along with scanZip()) open a zip, and add files from it. But from what i can tell, they are basically doing the same exact thing, which seems to be quite a pointless duplication of effort.
Does anyone have any ideas why it seems to be doing this?
Currently i have rewritten my zip handling code to handle a custom container format, and i have only implemented the searchPath() route.
I've not got files loading from the container yet (resource manager seems to think they do not exist, though they apear in the resource list) - though i'm getting closer to a solution ;)
About the author
#2
Some Ideas for future
The linux version of zip has a -e option which lets you set a password for the zip files. It would be good if resManager could make use of this feature ( programmer could hard code his password?) to protect textures and other artwork in the zips. Thus no need for a proprietory format.
Artists, level designers etc. may want to supply missions for a mod in a zip file which contains the mission file and art in the correct folders. resManager would then need to be able to identify and load these additional mission zips. Perhaps a naming convention where the mod might be mymod.zip and new missions might be mymod_1.zip, mymod_2.zip etc.
10/03/2004 (2:27 pm)
I tried making a zip of my mod folder and naming the zip the same as the mod folder but it does not load it.Some Ideas for future
The linux version of zip has a -e option which lets you set a password for the zip files. It would be good if resManager could make use of this feature ( programmer could hard code his password?) to protect textures and other artwork in the zips. Thus no need for a proprietory format.
Artists, level designers etc. may want to supply missions for a mod in a zip file which contains the mission file and art in the correct folders. resManager would then need to be able to identify and load these additional mission zips. Perhaps a naming convention where the mod might be mymod.zip and new missions might be mymod_1.zip, mymod_2.zip etc.
#3
If your mod folder is named mymod, then cd into that folder and create a zip from there called mymod.zip then move that zip file up one directory so that its in the same folder as the game executable.
One problem. I'm using Linux and the sound files all seem to get chopped short and some don't get loaded. Perhaps it's a memory shortage issue because the zip gets unzipped in memory. The sound is not an issue when not using the zip. I have not checked this in windows yet
10/03/2004 (2:47 pm)
OK, it does read from the zip. I found a post by Harold Labrat Brown which solved it.If your mod folder is named mymod, then cd into that folder and create a zip from there called mymod.zip then move that zip file up one directory so that its in the same folder as the game executable.
One problem. I'm using Linux and the sound files all seem to get chopped short and some don't get loaded. Perhaps it's a memory shortage issue because the zip gets unzipped in memory. The sound is not an issue when not using the zip. I have not checked this in windows yet
#4
For encryption, you'd need to add an extra flag to the ResourceObject, to indicate "Encrypted" status. You could then check for this flag in openStream(), where it opens a FilterStream on the zip.
If you wanted to have individual encryption keys, you'd probably have to override load() and openStream() so that you set and pass along the key to your appropriate crypt routines.
If you just wanted global encryption keys, say for the individual .zip's, you could just add an extra variable onto ZipAggregate, and add extra functions (to say, the ResourceManager) to set the key.
I am unsure how the zip format works in the encryption area though.
Beware though that there appears to be no standard for zip encryption between compressors (or so i've heard).
As for your sound getting chopped issue, the zip code uses buffers of a fixed size to unzip the data. It reads in a block of compressed data, then keeps decompressing to another buffer, which is passed onto the read(). When it has read all it can from the current block of compressed data, it reads another block, etc.
10/03/2004 (10:34 pm)
Oziman,For encryption, you'd need to add an extra flag to the ResourceObject, to indicate "Encrypted" status. You could then check for this flag in openStream(), where it opens a FilterStream on the zip.
If you wanted to have individual encryption keys, you'd probably have to override load() and openStream() so that you set and pass along the key to your appropriate crypt routines.
If you just wanted global encryption keys, say for the individual .zip's, you could just add an extra variable onto ZipAggregate, and add extra functions (to say, the ResourceManager) to set the key.
I am unsure how the zip format works in the encryption area though.
Beware though that there appears to be no standard for zip encryption between compressors (or so i've heard).
As for your sound getting chopped issue, the zip code uses buffers of a fixed size to unzip the data. It reads in a block of compressed data, then keeps decompressing to another buffer, which is passed onto the read(). When it has read all it can from the current block of compressed data, it reads another block, etc.
#5
As an update, I tried the zip method in windows using winzip and another package, but they both give the same in-game sound problem as in Linux. I also installed the latest zlib code which is quite a few versions ahead of torque's version, updated the targets file, re-compiled the engine, but the problem remains.
This tells me the bug is in resManager.cc and the associated zip functions which access the zlib library.
10/04/2004 (11:07 pm)
James, thanks for the ideas on how to do the encription. I will look into it as I get time ( never enough)As an update, I tried the zip method in windows using winzip and another package, but they both give the same in-game sound problem as in Linux. I also installed the latest zlib code which is quite a few versions ahead of torque's version, updated the targets file, re-compiled the engine, but the problem remains.
This tells me the bug is in resManager.cc and the associated zip functions which access the zlib library.
Torque 3D Owner Robert Blanchet Jr.
But I do agree with you. The zip archive functionality should be more streamlined, as well as loading modifications. If you have any ideas please be sure to voice them. This is one area that will be getting looked at as we move past Torque 1.3.