Atlas texture generation crash
by ENAK · in Torque Game Engine Advanced · 04/27/2007 (12:06 pm) · 45 replies
Hello,
A new guy here - and already with some problems. :) Weird. :)
Anyway, this is the problem. I'm trying to create a new atlas terrain + texture. I generated some terrain using L3DT, exported to raw, then created everything everything else up to texture...
This is what I have:
Some terrain in raw with 2048x2048 resolution (+1 of course)
The texture jpg for it with 8192x8192 resolution.
I'm following what's in the wiki. The atlasOldGenerateChunkFileFromRaw16, importOldAtlasCHU work fine.
But when I try to generate the texture - TGEA crashes to windows.
atlasGenerateTextureTOCFromLargeJPG("terrain_water_demo/texture.jpg", 8, "terrain_water_demo/texture.atlas");
I've tried fiddling with the tree depth but no value works.
The crash occurs on the purge method in the atlasImportLargeImage.cpp after it does a couple of tiles... (Not always the same number of them...)
If I comment this purge - it goes all the way, but then crashes on
So... Any ideas? This is TGEA 1.0.1, compiled with VS 2005 SP1.
...help... :(
I've also tried using a smaller texture or to generate from tiles - always crashes the same way. :(
Here's a more or less useless tail of the log.
A new guy here - and already with some problems. :) Weird. :)
Anyway, this is the problem. I'm trying to create a new atlas terrain + texture. I generated some terrain using L3DT, exported to raw, then created everything everything else up to texture...
This is what I have:
Some terrain in raw with 2048x2048 resolution (+1 of course)
The texture jpg for it with 8192x8192 resolution.
I'm following what's in the wiki. The atlasOldGenerateChunkFileFromRaw16, importOldAtlasCHU work fine.
But when I try to generate the texture - TGEA crashes to windows.
atlasGenerateTextureTOCFromLargeJPG("terrain_water_demo/texture.jpg", 8, "terrain_water_demo/texture.atlas");
I've tried fiddling with the tree depth but no value works.
The crash occurs on the purge method in the atlasImportLargeImage.cpp after it does a couple of tiles... (Not always the same number of them...)
for(S32 j=0; j<tileCount; j++)
{
// Allocate a chunk...
AtlasTexChunk *atc = new AtlasTexChunk;
Con::printf(" Point 1 %d/%d",j,tileCount);
atc->mFormat = AtlasTexChunk::FormatPNG;
// Copy from the JPEG working space to the bitmap.
GBitmap *gb = new GBitmap(tileSize, tileSize);
for(S32 k=0; k<tileSize; k++)
dMemcpy(gb->getAddress(0, k), rows[k] + j*tileSize*3, 3*tileSize);
atc->bitmap = gb;
// Get the stub.
AtlasResourceTexStub *tStub = arttoc->getStub(treeDepth-1, Point2I(i,j));
// Store, then purge, the chunk.
arttoc->instateNewChunk(tStub, atc, true);
tStub->purge(); // <--- crash!
}If I comment this purge - it goes all the way, but then crashes on
arttoc->generate(RectI(0, 0, tileCount, tileCount));
So... Any ideas? This is TGEA 1.0.1, compiled with VS 2005 SP1.
...help... :(
I've also tried using a smaller texture or to generate from tiles - always crashes the same way. :(
Here's a more or less useless tail of the log.
atlasGenerateTextureTOCFromLargeJPEG - Successfully opened JPEG 'terrain_water_demo/texture.jpg' for reading. atlasGenerateTextureTOCFromLargeJPEG - Image is 8192px (hopefully square!). atlasGenerateTextureTOCFromLargeJPEG - Tree depth of 5 mandates 16 tiles on a side. atlasGenerateTextureTOCFromLargeJPEG - Therefore, tiles are 512px wide. AtlasDeferredFile::ensureStreamWritable - switching file into writable mode... This may cause a crash if you have other copies of Torque accessing it! atlasGenerateTextureTOCFromLargeJPEG - Atlas file 'terrain_water_demo/texture.atlas' created! atlasGenerateTextureTOCFromLargeJPEG - Initialized Atlas threads, now importing texture data... atlasGenerateTextureTOCFromLargeJPEG - Allocating ~12288 kb of JPEG decompression working area... atlasGenerateTextureTOCFromLargeJPEG - Generating 512px by 512px leaf tiles...
#2
As for myself and Jeremiah's issue we are using blended terrains not unique terrains like the first poster appears to be doing. Right now crashing on the png opacity map tile import section of the process, right when it tries to import the first tile. We have tried it with different tile count (all 512x512 tiles just some regular res and sometimes hi-res tile counts) and I spent a good thirteen hours trying everything I could think of to get it to work including using some of our older blended terrain generation scripts that did work in the past with no luck.
Any help would be obliged.
04/30/2007 (4:52 am)
ENAK if your texture tree depth is really high and I suspect that that might be part of your problem if you can try compiling it with a tree depth in the 3-5 range although without being able to see your terrain generation commands that's just a ballpark generic figure, (post the commands up if you can) and if nothing else a tree depth that high will create some major load time issues not to mention a huge file size. As for myself and Jeremiah's issue we are using blended terrains not unique terrains like the first poster appears to be doing. Right now crashing on the png opacity map tile import section of the process, right when it tries to import the first tile. We have tried it with different tile count (all 512x512 tiles just some regular res and sometimes hi-res tile counts) and I spent a good thirteen hours trying everything I could think of to get it to work including using some of our older blended terrain generation scripts that did work in the past with no luck.
Any help would be obliged.
#3
More details in this thread [sorry for the double - didn't notice it before]
http://www.garagegames.com/mg/forums/result.thread.php?qt=60568
What I did notice is that there seems to be something wrong with the part responsible for loading the bitmaps. Maybe the gBitmap class itself has some error somewhere. Because after the pointer assignment (bitmap pointer in stub) the stub's chunk pointer can no longer be deleted (unallocated block error)... Weird. :(
04/30/2007 (5:15 am)
I tried all kinds of values for the tree depth. Crashed every time. Even on a tiny one leaf 256x256 texture it also crashed... More details in this thread [sorry for the double - didn't notice it before]
http://www.garagegames.com/mg/forums/result.thread.php?qt=60568
What I did notice is that there seems to be something wrong with the part responsible for loading the bitmaps. Maybe the gBitmap class itself has some error somewhere. Because after the pointer assignment (bitmap pointer in stub) the stub's chunk pointer can no longer be deleted (unallocated block error)... Weird. :(
#4
04/30/2007 (6:36 am)
Ya thats the same error we are getting (Unallocated Block)
#5
04/30/2007 (10:48 am)
If I comment out the purge command in atlasGenerateTextureTOCFromTiles, it will make it further, but still crash when it tries to save out the Chunk, so it would seem multilayer support additions in 1.01 are completely unstable or something else is broken and this just made it "visible"
#6
I would try this:
A change was made to support more than 4 channels (ie, multiple bitmaps in a texchunk, which is required for blending and some other things) and it looks like not every import path got updated.
04/30/2007 (10:49 am)
There is an update to TGEA in the pipe, which is likely to fix the problems you're experiencing.I would try this:
// Copy from the JPEG working space to the bitmap.
GBitmap *gb = new GBitmap[1];
gb->allocateBitmap(tileSize, tileSize);
for(S32 k=0; k<tileSize; k++)
dMemcpy(gb->getAddress(0, k), rows[k] + j*tileSize*3, 3*tileSize);
atc->layerCount = 1;
atc->bitmap = gb;A change was made to support more than 4 channels (ie, multiple bitmaps in a texchunk, which is required for blending and some other things) and it looks like not every import path got updated.
#7
That would seem to apply to the other function (TOCfromLargeJPEG), any information about the crash within atlasGenerateTextureTOCFromTiles, dealing with either the purge crashing or, if thats commented out, crashing when it goes to write out the chunk
04/30/2007 (11:13 am)
@Ben,That would seem to apply to the other function (TOCfromLargeJPEG), any information about the crash within atlasGenerateTextureTOCFromTiles, dealing with either the purge crashing or, if thats commented out, crashing when it goes to write out the chunk
#8
Is it taking the getNumberChannels() == 0 path while importing?
It's likely to be the same issue but only some work with the debugger is going to reveal the true nature of your problem!
04/30/2007 (11:49 am)
Did you check the code to be sure that it's properly setting the layerCount and that atc->bitmap is an array allocation rather than a single item?Is it taking the getNumberChannels() == 0 path while importing?
It's likely to be the same issue but only some work with the debugger is going to reveal the true nature of your problem!
#9
However...
1) Tiles still don't work of course...
and
2) Now there another unallocated block crash in the console function: atlasGenerateUniqueTerrain (not sure about the blended one)
This is in the atlasConsoleGenerator.cpp file around the
Inside that method it crashes on:
04/30/2007 (11:53 am)
Thanks, that seems to have done the trick for texture generation from large JPEG.However...
1) Tiles still don't work of course...
and
2) Now there another unallocated block crash in the console function: atlasGenerateUniqueTerrain (not sure about the blended one)
This is in the atlasConsoleGenerator.cpp file around the
Con::printf(" o Copying texture chunks...");
arttoc->copyChunksToTOC(newArtToc);Inside that method it crashes on:
origArgs->purge();
#10
04/30/2007 (12:02 pm)
What does generateCopy look like in your copy of Torque? Mine looks like:AtlasTexChunk *AtlasTexChunk::generateCopy(S32 reformat)
{
AtlasTexChunk *atc = new AtlasTexChunk;
// Support changing the format as we go.
if(reformat > -1)
atc->mFormat = (TexFormat)reformat;
else
atc->mFormat = mFormat;
atc->layerCount = layerCount;
// We have to copy our layers...
if(bitmap)
{
atc->bitmap = new GBitmap[layerCount];
for(S32 layer=0; layer<layerCount; layer++)
{
//if(bitmap[layer].sgAlreadyConverted)
// bitmap[layer].convertToBGRx(true);
bitmap[layer].swizzle(GFX->getDeviceSwizzle());
atc->bitmap[layer] = bitmap[layer];
}
}
if(dds)
{
atc->dds = new DDSFile[layerCount];
for(S32 layer=0; layer<layerCount; layer++)
atc->dds[layer] = dds[layer];
}
return atc;
}
#11
So it's as in TGEA 1.0.1
I do have to say there's a glaring difference in the middle. ;)
That swizzle method... whatever that does. I don't have it in my gBitmap class... Also I don't have this getDeviceSwizzle method in GFXDevice...
04/30/2007 (12:14 pm)
I didn't change anything except what I was experimenting on earlier (now reverted to original)...So it's as in TGEA 1.0.1
AtlasTexChunk *AtlasTexChunk::generateCopy(S32 reformat)
{
AtlasTexChunk *atc = new AtlasTexChunk;
// Support changing the format as we go.
if(reformat > -1)
atc->mFormat = (TexFormat)reformat;
else
atc->mFormat = mFormat;
atc->layerCount = layerCount;
// We have to copy our layers...
if(bitmap)
{
atc->bitmap = new GBitmap[layerCount];
for(S32 layer=0; layer<layerCount; layer++)
{
if(bitmap[layer].sgAlreadyConverted)
bitmap[layer].convertToBGRx(true);
atc->bitmap[layer] = bitmap[layer];
}
}
if(dds)
{
atc->dds = new DDSFile[layerCount];
for(S32 layer=0; layer<layerCount; layer++)
atc->dds[layer] = dds[layer];
}
return atc;
}I do have to say there's a glaring difference in the middle. ;)
That swizzle method... whatever that does. I don't have it in my gBitmap class... Also I don't have this getDeviceSwizzle method in GFXDevice...
#12
For further debugging, I'd suggest putting a counter or something on AtlasTexChunk so you can identify specific allocations, then backtracking to see when the chunk with the id that is causing the crash is allocated.
Are you running your generation scripts from a main.cs with no other instructions? Running it from within a mission is definitely a bad idea.
This will get a pass as part of the 1.0.2 release; I won't have time to look at it in detail before then so if you want a solution sooner then the burden of debugging will be on the community; I'll gladly offer my opinions but I have a set of tasks that must get done before 1.0.2 happens so can't burn days on it right now.
04/30/2007 (1:13 pm)
The swizzle stuff isn't relevant, just playing with changing how we change bitmap formats.For further debugging, I'd suggest putting a counter or something on AtlasTexChunk so you can identify specific allocations, then backtracking to see when the chunk with the id that is causing the crash is allocated.
Are you running your generation scripts from a main.cs with no other instructions? Running it from within a mission is definitely a bad idea.
This will get a pass as part of the 1.0.2 release; I won't have time to look at it in detail before then so if you want a solution sooner then the burden of debugging will be on the community; I'll gladly offer my opinions but I have a set of tasks that must get done before 1.0.2 happens so can't burn days on it right now.
#13
This goes on before anything gets initiated. The terrain and it's scale are large but it crashes also on small stuff during copying of the texture chunks for unique terrain (first chunk actually, on the purge method as I mentioned above).
So... TGEA 1.0.2... Any time line for that release? I kinda need this to work. ;)
04/30/2007 (1:46 pm)
I've added a new command line param to generate the atlas stuff... it goes like this:case "-genter":
enableWinConsole(true);
setLogMode(6);
atlasOldGenerateChunkFileFromRaw16("terrain_water_demo/heightfield.raw",2048,1.0,1.0/32.0,"terrain_water_demo/height.chu",1.0,7);
importOldAtlasCHU("terrain_water_demo/height.chu","terrain_water_demo/height.atlas");
atlasGenerateTextureTOCFromLargeJPEG("terrain_water_demo/texture.jpg",5,"terrain_water_demo/texture.atlas");
atlasGenerateUniqueTerrain("terrain_water_demo/output.atlas","terrain_water_demo/height.atlas","terrain_water_demo/texture.atlas");
echo("Done!!!");
$argUsed[%i]++;This goes on before anything gets initiated. The terrain and it's scale are large but it crashes also on small stuff during copying of the texture chunks for unique terrain (first chunk actually, on the purge method as I mentioned above).
So... TGEA 1.0.2... Any time line for that release? I kinda need this to work. ;)
#14
No announced release date yet for 1.0.2, and I'm not gonna make one up on the spot - against company policy. :)
Meanwhile, if you need it to work, your best bet is to debug it yourself. Even if 1.0.2 was coming out tomorrow there's no guarantee that it'll resolve the exact problem you're experiencing, so if you're blocked on this issue you should roll up your sleeves and fix it. Like I said - I'm more than happy to give advice on solving the problem and interpret results for you, I just don't have the luxury of switching gears off of my current tasks to spend the day or two it would take to fix this.
04/30/2007 (2:03 pm)
What if you just do the TOCFromLargeJPEG by itself? At least that way we can be sure the problem isn't due to interaction between these things.No announced release date yet for 1.0.2, and I'm not gonna make one up on the spot - against company policy. :)
Meanwhile, if you need it to work, your best bet is to debug it yourself. Even if 1.0.2 was coming out tomorrow there's no guarantee that it'll resolve the exact problem you're experiencing, so if you're blocked on this issue you should roll up your sleeves and fix it. Like I said - I'm more than happy to give advice on solving the problem and interpret results for you, I just don't have the luxury of switching gears off of my current tasks to spend the day or two it would take to fix this.
#15
Oh well - I'll try my luck with the debugger tomorrow again (Cyprus... It's midnight ;) ).
Thanks for the first fix.
Just a quick question. It's not crashing for you guys at the Unique Terrain generation function?
04/30/2007 (2:07 pm)
I tried doing all the stages separately before also... No use... :(Oh well - I'll try my luck with the debugger tomorrow again (Cyprus... It's midnight ;) ).
Thanks for the first fix.
Just a quick question. It's not crashing for you guys at the Unique Terrain generation function?
#16
I'll do some hunting this evening to verify path it is taking, although it crashes differently in both cases.. One, if i comment out a purge, itll go on thru and then crash on saving out... Otherwise, itll crash at the purge it seems, unless something is getting corrupted or fouled, and the purge is "catching" it and crashing earlier.
04/30/2007 (3:51 pm)
@Ben, I'll do some hunting this evening to verify path it is taking, although it crashes differently in both cases.. One, if i comment out a purge, itll go on thru and then crash on saving out... Otherwise, itll crash at the purge it seems, unless something is getting corrupted or fouled, and the purge is "catching" it and crashing earlier.
#17
Thanks for your willingness to help solve this problem!
@ENAK: It doesn't when we export via L3DT which uses almost the same code. As for the unique case, I don't have opportunity to test that right now.
@Jeremiah: If you compile with debug guards on then you can call Memory::validate(); before and after suspect parts of the code, and it'll break if something's writing over the guard values at the beginning/end of allocated memory blocks.
My guess is some code somewhere is not taking into account that the AtlasTexChunk::bitmaps member stores an array now, and is thus clobbering important data. So you'd look for places that were calling new without an array count.
04/30/2007 (4:28 pm)
Hey guys,Thanks for your willingness to help solve this problem!
@ENAK: It doesn't when we export via L3DT which uses almost the same code. As for the unique case, I don't have opportunity to test that right now.
@Jeremiah: If you compile with debug guards on then you can call Memory::validate(); before and after suspect parts of the code, and it'll break if something's writing over the guard values at the beginning/end of allocated memory blocks.
My guess is some code somewhere is not taking into account that the AtlasTexChunk::bitmaps member stores an array now, and is thus clobbering important data. So you'd look for places that were calling new without an array count.
#18
04/30/2007 (7:04 pm)
The crash with purge is tied to SAFE_DELETE_ARRAY(bitmap) in AtlasTexChunk::~AtlasTexChunk()Ascension_DEBUG.exe!AtlasTexChunk::~AtlasTexChunk() Line 20 + 0x5c bytes C++ Ascension_DEBUG.exe!AtlasTexChunk::'scalar deleting destructor'() + 0x2b bytes C++ Ascension_DEBUG.exe!AtlasResourceStub<AtlasTexChunk>::purge() Line 98 + 0x37 bytes C++ Ascension_DEBUG.exe!catlasGenerateTextureTOCFromTiles(SimObject * __formal=0x00000000, int argc=5, const char * * argv=0x00e41868) Line 103 C++
+ bitmap 0x04becbb4 {sgAlreadyConverted=false sBitmapIdSource=0 internalFormat=GFXFormatR8G8B8 ...} GBitmap *
+ this 0x04becc54 {mFormat=FormatPNG layerCount='' bitmap=0x04becbb4 ...} AtlasTexChunk * const
#19
The purge to comment out is in this place in atlasResourceTOC.cpp:
The generated file seems fine - it loads up correctly into the engine, the textures don't seem messed up...
05/01/2007 (5:52 am)
The crash in atlasGenerateUniqueTerrain can temporarily be solved by commenting out the purge in the copyChunksToTOC function. Maybe a good idea to make a copy of this method as it creates of course a nasty memory leak and it's used elsewhere (hopefully doesn't crash there).The purge to comment out is in this place in atlasResourceTOC.cpp:
toc->instateNewChunk(args, origArgs->mChunk->generateCopy(reformat), true);
args->purge();
// origArgs->purge();The generated file seems fine - it loads up correctly into the engine, the textures don't seem messed up...
#20
These fixes are in for the next TGEA update. Meanwhile you should have all the info you need in this thread to resolve immediate issues.
05/11/2007 (3:19 pm)
I finally found some time to test and the large JPEG path works fine (with the fixes I posted in this thread). I also tested the tile import and found a memory management issue, which was resolved by the following change:dSprintf(buff, 512, tileMask, imgX, imgY);
GBitmap *bmp = GBitmap::load(buff);
if(!bmp)
{
Con::errorf("atlasGenerateTextureTOCFromTiles - unable to open tile '%s'! Aborting!", buff);
return;
}
// Create a chunk to store this data in.
AtlasTexChunk *atc = new AtlasTexChunk;
atc->mFormat = (AtlasTexChunk::TexFormat)outFormat;
atc->layerCount = 1;
// Copy the loaded bitmap into our array of a single bitmap.
atc->bitmap = new GBitmap[1];
// Note that this copies the _pointers_ so we don't have to worry
// about leaking bmp's resources - they're transferred to atc->bitmap[0].
atc->bitmap[0] = *bmp;
// Get the stub.
AtlasResourceTexStub *tStub = arttoc->getStub(treeDepth-1, Point2I(x,y));
// And store the chunk.
arttoc->instateNewChunk(tStub, atc, true);
// Purge it once we're done so we don't use gigs of ram!
tStub->purge();These fixes are in for the next TGEA update. Meanwhile you should have all the info you need in this thread to resolve immediate issues.
Torque Owner Jeremiah Fulbright
Any information about this Ben?