Game Development Community

Crash in Standard Zip Encryption Support resource

by Stephane Conde · in Torque Game Engine · 10/18/2006 (7:00 pm) · 21 replies

First, the background...

I posted about a crash bug I am having with encrypted zips in the Standard Zip Encryption Support resource: http://www.garagegames.com/index.php?sec=mg&mod=resource&page=view&qid=9896

To summarize what that post says, I am getting a crash in ZipLocalFileHeader::readFromStream(Stream &).

As an update to that post, the crash is happening specifically on the following line:

bool success = io_rStream.read(&m_header.headerSig);

The 'read' it is doing is somehow causing it to crash. Possibly the most strange part of all of this is that the issue seems to disappear when I try to debug the exact same executable. If I run the identical executable through XCode's debugger, the problem vanishes! I'm really not sure what to make of that and it is making the problem that much more difficult to fix. I was able to narrow the line down using 'print' statements, but I unfortunately cannot get any more information than that.

Does anyone have any ideas as to what might be causing this issue? For more information (including the crash log) see the previous post...

Thank you very much in advance for any help you might be able to offer.

Stephane
Page «Previous 1 2
#1
10/18/2006 (10:05 pm)
Just an update... I tried creating an unencrypted zip file and it worked out just fine... so its got to be something with the encrypted zip resource...

How does everyone else out there create encrypted zips for their Mac builds? Do you create the zip file on your windows machine and then just copy it over?

Any ideas?
#2
10/18/2006 (11:23 pm)
Yet another update to this one...

The crash is actually happening one line before where I thought it was happening (it was difficult to tell without being able to debug the thing...)

The crash is happening on this line:

U32 initialPosition = io_rStream.getPosition();

I am really surprised that no one else is experiencing this issue...
#3
10/19/2006 (12:11 am)
Hmm... I guess it was asserting in Debug mode, but on the Mac the asserts just seem to go to the log file...

Here is the assert in debug mode:
Fatal: (/Users/sconde/Documents/build_current/engine/core/fileStream.cc @ 46) FileStream::setPosition: the stream isn't open
#4
10/19/2006 (1:29 am)
Pretty much what it says, you need to open the stream before you access it.
I was under the impression though, that the standard ZIP encryption resource was not endian safe.
#5
10/19/2006 (10:42 am)
Hey Stefan,

Thanks for the reply...

I'm pretty sure the stream is actually open... Especially since it read the entire menu system out of the zip file... Also, things work fine when I run the game through the Debugger... so something very strange is going on...

I tried creating the zip on the Macintosh itself, but it still has the exact same problem.

According to the Standard Zip Encryption Support resource, it works on Macs. Venture Africa has been using that exact resource and they have a Mac version, so I don't know why its not working for me...

Any other thoughts on this one?
#6
10/19/2006 (10:53 am)
Quote:
I'm pretty sure the stream is actually open... Especially since it read the entire menu system out of the zip file... Also, things work fine when I run the game through the Debugger... so something very strange is going on...

Could you expand on this? *What* debugger? :) Your debug build or the IDE debugger? Sounds like a relative path issue somewhere.
#7
10/19/2006 (11:18 am)
Sorry... The GDB Debugger through XCode 2.4...

Yeah, that's what I would assume too, but its running the same executable so I don't know how it can be a path issue...
#8
10/20/2006 (11:28 pm)
Okay, a little further information on this one...

I added some code to the top of the ZipLocalFileHeader::readFromStream(Stream &) function to find out the 'status' of the stream. Apparently, the status of the stream is: Stream::UnknownError. Has anyone seen that before? Is that indicative of anything?

Any help would be greatly appreciated.

Thanks!

Stephane
#9
10/21/2006 (1:02 am)
Ugh... yet another update on this...

It seems like the zip file isn't being opened for some reason...

The reason the stream is being set to UnknownError is that the fopen(...) call inside of File::open in MacCarbFileio.cc is not properly opening the file. Why would running the game normally cause it to not open the zip file at a certain point but running it through the debugger cause it to work perfectly fine?

Grr... this is a tough one...

Stephane
#10
10/21/2006 (1:36 am)
Okay... another update...

Doing strerror(errno) in File::open just after it makes the fopen(...) call gives me a "Too many open files" error. What the heck is that about?

Stephane
#11
10/21/2006 (2:37 am)
Alright... so it appears that Mac's have a hard limit of 256 files open at a time... And our game apparently goes above this limit (also apparently only when using encrypted zips outside of a debugger). Calling 'ulimit -n 2000' just before running the game stops it from crashing...

Obviously this isn't a permanent solution...

Any suggestions for why the limit might be exceeded only when using encrypted zips?

Thanks,

Stephane
#12
10/21/2006 (2:43 am)
Does it work in a debug build alone, or do you have to actually run the debugger ontop of the application? If the former, then it is probably something simple.
#13
10/21/2006 (3:35 am)
Thank you for the reply Stefan... I appreciate the help.

No, it doesn't work in debug build...

Here's a little chart for you Stefan... to clear up any confusion...
normal               debugger
release build    doesn't work                works
                   
debug build      doesn't work                works
debug/release builds with increased file limit works for both normal and debugger.


Okay, another update as well: I've done some more debugging (added a static int to the FileStream class to keep track of the current number of open files) and every time you open a file from an encrypted zip it leaves a file open which ends up crashing the game on the Mac when you get to 256. When using un-encrypted zips, no zips or running through the debugger this doesn't happen.
#14
10/21/2006 (4:02 am)
Oh man... I'm an idiot... the issue was right there in front of me the whole time!

It looks like its a bug in the Torque engine. Torque was never designed to have more than one level of 'FilterStream'.

Standard zips look like this:

ZipSubRStream->FileStream or ResizeFilterStream->FileStream

But encrypted zips look like this:

ZipCryptRStream->ZipSubRStream->FileStream or ZipCryptRStream-> ResizeFilterStream->FileStream

So the encrypted zip streams have two levels of 'FilterStreams'... but the ResourceManager only deals with one level, so it was not properly closing the streams after they'd been opened.

Here is the fixed ResourceManager::closeStream(...) function:
void ResManager::closeStream(Stream* stream)
{
   FilterStream *currentStream, *nextStream;
	
   // Try to cast the stream to a FilterStream
   nextStream = dynamic_cast<FilterStream*>(stream);
	
   // While the nextStream is valid (meaning it was successfully cast to a FilterStream)
   while (nextStream)
   {
      // Point currentStream to nextStream
      currentStream = nextStream;
      // Point stream to the Stream contained within the current FilterStream
      stream = currentStream->getStream();
      // Detach the current FilterStream from the Stream contained within it
      currentStream->detachStream();
      // Try to cast the stream (which was already contained within a FilterStream) to a FilterStream
      nextStream = dynamic_cast<FilterStream*>(stream);
      // Delete the FilterStream that was wrapping stream
      delete currentStream;
   }

/*
   FilterStream* subStream = dynamic_cast<FilterStream*>(stream);
   if (subStream)
   {
      stream = subStream->getStream();
      subStream->detachStream();
      delete subStream;
   }
*/
   delete stream;
}

This new function should be able to deal with any number of 'FilterStreams' being used.

Thanks for the help Stefan.

Hope this helps others!

Stephane
#15
10/21/2006 (5:21 am)
I find it suprising that it has not been mentioned before or even fixed. This should definatly go into HEAD. Thanks for making the fix public.
#16
10/21/2006 (11:00 am)
Hmm. Good catch. That's a pretty lame bug. I'll see about getting it into HEAD.

T.
#17
10/21/2006 (3:52 pm)
My apologies... I was pretty tired last night...

Here's a much more concise fix:

void ResManager::closeStream(Stream* stream)
{
   // Try to cast the stream to a FilterStream
   FilterStream* subStream = dynamic_cast<FilterStream*>(stream);
	
   // While the sub-stream is valid (meaning it was successfully cast to a FilterStream)
   while (subStream)
   {
      // Point stream to the Stream contained within the current FilterStream
      stream = subStream->getStream();
      // Detach the sub-stream FilterStream from the Stream contained within it
      subStream->detachStream();
      // Delete the FilterStream that was wrapping stream
      delete subStream;
      // Try to cast the stream (which was already contained within a FilterStream) to a FilterStream
      subStream = dynamic_cast<FilterStream*>(stream);
   }

/*
   FilterStream* subStream = dynamic_cast<FilterStream*>(stream);
   if (subStream)
   {
      stream = subStream->getStream();
      subStream->detachStream();
      delete subStream;
   }
*/
   delete stream;
}

Stephane
#18
10/22/2006 (12:29 pm)
This is EXACTLY what was plaugine me 8 months ago. Wasnt able to get much help back then on it and I certainly didnt/dont have the coding skills to figure it out.. I cant say "OMG THANK YOU" enough Stephane.

*does the happy dance*

Thanks ;)
#19
10/22/2006 (2:14 pm)
Hey Flybynight,

I didn't see your posts about this problem anywhere. Make sure when you have issues like this to at least make a post about it, even if you can't or don't want to fix it. It will not only alert people to a potential problem, but it will also help others when they go to fix the issue. I was convinced that the issue was something I was doing wrong for the longest time since no one else had reported the same issue.

Glad you found the fix helpful.

Stephane
#20
05/25/2007 (12:38 pm)
@Stephane: Hey awesome job on this fix! Saved me a lot of work!!! ;)

I was wondering what zip program you ended up using to create your zip's (for the Mac, that is).
Page «Previous 1 2