Atlas Improvement/Fixing Thread

#1 // This improvement is included in TGEA 1.7.0 final

Improvement: AtlasChunk::readFromStream Optimization

TGEA version: 1.7 Beta 2

Description

Atlas input passes three stages of processing: first by AtlasDeferredFile, then by AtlasChunk, and then by the actual chunk type implementation. The implementation in AtlasChunk is generic to allow data to come from any stream. This, however, adds unecessary overhead when data is already in memory. In this case, AtlasDeferredFile will move from file to memory, AtlasChunk will copy from memory to memory, and the actual chunk type will then process from memory to memory. Very inefficient.

The following replacement for AtlasChunk::readFromStream improves on this.

Source Code

atlasChunk.cpp@51

bool AtlasChunk::readFromStream(AtlasChunk *ac, Stream *s)
{
   // First, read the size and allocate a buffer.
   
   // Let's do a sentinel check.
   U32 sent1;
   s->read(&sent1);
   AssertISV(sent1 == MakeFourCC('A', 'T', 'S', 'P'), "AtlasChunk::readFromStream - (sent1) invalid chunk master sentinel!");

   s->read(&ac->mChunkSize);

   // And now validate the chunk-type-sentinel.
   U32 sent2;
   s->read(&sent2);
   AssertISV(sent2 == ac->getHeadSentinel(), "AtlasChunk::readFromStream - (sent2) invalid chunk head sentinel!");

   // Get the chunk's data.  If it's already in memory, just reuse it.
   // Otherwise read it from the stream.

   bool isMemStream = false;
   U8 *data;
   U32 dataSize = ac->mChunkSize;
   
   if( dynamic_cast< MemStream* >( s ) )
   {
	   MemStream* memStream = ( MemStream* ) s;
	   U32 currentPos = memStream->getPosition();
	   isMemStream = true;
	   data = &( ( U8* ) memStream->getBuffer() )[ currentPos ];
	   memStream->setPosition( currentPos + dataSize );
   }
   else
   {
	   data = new U8[ dataSize ];

	   // Read next chunksize bytes into the buffer for later processing.
	   s->read( dataSize, data);
   }

   // Check end sentinel.
   U32 sent3;
   s->read(&sent3);
   AssertISV(sent3 == ac->getTailSentinel(), "AtlasChunk::readFromStream - (sent3) invalid chunk tail sentinel!");

   // And tell the chunk to read from that buffer...
   MemStream dataStream( dataSize, data );
   ac->read(&dataStream);

   // Clean up memory.
   if( !isMemStream )
	   delete[] data;

   // All done!
   return true;
}

#2 //EDIT: 04/23/08 minor fix
// This improvement is included in TGEA 1.7.0 final

Improvement: Make detail texturing optional
TGEA Version: 1.7 Beta 2

Description

Atlas currently forces you to use detail mapping which may not be to everyone's taste (I don't like it). The following set of simple changes will disable detail mapping if the "detailTex" property of an Atlas instance is empty. Since detail textures are simply blended onto the terrain, this change is simple to do.

Source Code

In AtlasInstance::AtlasInstance@62 change

mDetailTexFileName = StringTable->insert("terrain_water_demo/data/terrains/details/detail1");

to

mDetailTexFileName = 0;

In AtlasInstance::onAdd@259 change

mDetailTex.set(mDetailTexFileName,&GFXDefaultStaticDiffuseProfile);

to

//EDIT: fixed

if( mDetailTexFileName && mDetailTexFileName[ 0 ] )
		 {
			if( !mDetailTex.set( mDetailTexFileName, &GFXDefaultStaticDiffuseProfile ) )
			{
				Con::warnf( "AtlasInstance2::onAdd - could not load detail map; disabling detail mapping" );
				mBatcher.isDetailMappingEnabled( false );
			}
		 }
		 else
			 mBatcher.isDetailMappingEnabled( false );

Add the following code to the interface of AtlasClipMapBatcher

private:
   bool mIsDetailMappingEnabled;
public:
   bool isDetailMappingEnabled()
   {
	   return mIsDetailMappingEnabled;
   }
   void isDetailMappingEnabled( bool value )
   {
	   mIsDetailMappingEnabled = value;
   }

Add this to AtlasClipMapBatcher::AtlasClipMapBatcher:

mIsDetailMappingEnabled = true;

In AtlasClipMapBatcher::queue@89 change:

if(nearDistance < mDetailMapFadeEndDistance)
      mDetailList.push_back(rn);

to

if( isDetailMappingEnabled() && nearDistance < mDetailMapFadeEndDistance )
		   mDetailList.push_back(rn);

And finally in AtlasClipMapBatcher::render@387 change

renderDetail();

to

if( isDetailMappingEnabled() )
	   renderDetail();

#3 Fix: Crash in atlasGenerateTextureTOCFromLargeJPEG
TGEA Version: 1.7.0 final

Description

The following fixes a bug in atlasGenerateTextureTOCFromLargeJPEG that invariably crashes the engine due to improperly setting up texture chunks.

Source Code

In atlas/editor/atlasImportLargeImage.cpp@283 replace

// Copy from the JPEG working space to the bitmap.
         GBitmap *gb = new GBitmap(tileSize, tileSize);
         for(S32 k=0; k<tileSize; k++)
            dMemcpy(gb->getAddress(0, k), rows[k] + j*tileSize*3, 3*tileSize);
         atc->bitmap = gb;

with

// Copy from the JPEG working space to the bitmap.
	 atc->bitmap = new GBitmap[ 1 ];
	 atc->layerCount = 1;
         GBitmap *gb = &atc->bitmap[ 0 ];
	 new ( gb ) GBitmap( tileSize, tileSize );
         for(S32 k=0; k<tileSize; k++)
            dMemcpy(gb->getAddress(0, k), rows[k] + j*tileSize*3, 3*tileSize);

#4 //EDIT: Improvement (the previous version was ineffective to the point that it was useless)

Fix: Pipeline congestion in high throughput situations
TGEA Version: 1.7.0 final

Description

For maps with large unique textures or dense meshes, Atlas may wind up amassing outdated load requests in its input pipeline. This gets most noticeable in that part or all of the clip map detail levels stop updating and that Torque begins to consume large quantities of memory. If the camera remains still, the pipeline will clear out and the clip map will update again.

The following fix prevents congestion of the pipeline by allowing only a certain amount of requests to pass through at any one time and keeping all other requests in the first pipeline stage where they are still cancellable.

Source Code

In AtlasFile::enqueueNextPendingLoad (atlas/core/atlasFile.cpp@485) add the following code:

U32 deserializerLoad = mPendingDeserializeQueue.lockVector().size();
   mPendingDeserializeQueue.unlockVector();
   if( deserializerLoad > 10 || ( deserializerLoad > 5 && mFile.hasPendingIO() ) )
   {
	   PROFILE_END();
	   return;
   }

#5 Fix: Minor memory leak in AtlasGeomChunk::copyToDiscreteMesh
TGEA Version: 1.7.0 final

Description

There is a minor memory leak in AtlasGeomChunk::copyToDiscreteMesh that will cause the buffers allocated to the mesh to not be freed later on.

Source Code

Add the following code before the return statement of the method AtlasGeomChunk::copyToDiscreteMesh in atlas/resource/atlasGeomChunk.cpp:

adm->mOwnsData = true;

#6 //EDIT: code modified to fix clip map jumping

TEMPORARY Fix: Workaround to raycasting issues
TGEA Version: 1.7.0 final

Description

Atlas currently shows issues with some collision data that will cause raycasting misses against terrain geometry. This will most importantly result in recentering requests not being issued to the clipmap.

The following code is a temporary workaround that does not solve the underlying problem. I will substitute this with a real fix once I have one.

Source Code

This code works by adding a new ray collision debug level to Atlas that does raycasting against raw mesh geometry. You can then either set the level from the console or just default the engine to the level for the time being.

In atlas/runtime/atlasInstance2.h@102 add:

RayCollisionDebugToMesh		= 4, ///< Let raycasting test against raw geometry data rather than the collision mesh.

Then in atlas/resource/atlasGeomCollision.cpp@104 add:

else if( AtlasInstance::smRayCollisionDebugLevel == AtlasInstance::RayCollisionDebugToMesh )
   {
	   bool haveHit = false;
	   U32 currentIdx = 0;
	   U32 numIdx = mChunk->mIndexCount;
	   F32 bestT = F32_MAX;
	   U32 bestTri = -1;//EDIT: previously was U16
	   Point2F bestBary;

	   while( !haveHit && currentIdx < numIdx )
	   {
		   const Point3F& a = mChunk->mVert[ mChunk->mIndex[ currentIdx ] ].point;
		   const Point3F& b = mChunk->mVert[ mChunk->mIndex[ currentIdx + 1 ] ].point;
		   const Point3F& c = mChunk->mVert[ mChunk->mIndex[ currentIdx + 2 ] ].point;

		   F32     localT;
		   Point2F localBary;

		   // Do the cast, using our conveniently precalculated ray delta...
		   if(castRayTriangle(mRayStart, mRayDelta, a,b,c, localT, localBary))
		   {
			   if(localT < bestT)
			   {
				   // And it hit before anything else we've seen.
				   bestTri  = currentIdx;
				   bestT    = localT;
				   bestBary = localBary;

				   haveHit = true;
			   }
		   }

		   currentIdx += 3;
	   }

      // Fill in extra info for the hit.
      if(!haveHit)
         return false;

      // Calculate the normal, we skip that for the initial check.
      Point3F norm; // Hi norm!

      const Point3F &a = mChunk->mVert[mChunk->mIndex[bestTri+0]].point;
      const Point3F &b = mChunk->mVert[mChunk->mIndex[bestTri+1]].point;
      const Point3F &c = mChunk->mVert[mChunk->mIndex[bestTri+2]].point;

      const Point2F &aTC = mChunk->mVert[mChunk->mIndex[bestTri+0]].texCoord;
      const Point2F &bTC = mChunk->mVert[mChunk->mIndex[bestTri+1]].texCoord;
      const Point2F &cTC = mChunk->mVert[mChunk->mIndex[bestTri+2]].texCoord;

      // Store everything relevant into the info structure.
      info->t = bestT;

      const Point3F e0 = b-a;
      const Point3F e1 = c-a;

      info->normal = mCross(e1, e0);
      info->normal.normalize();

      // Calculate and store the texture coords.
      const Point2F e0TC = bTC-aTC;
      const Point2F e1TC = cTC-aTC;
      info->texCoord = e0TC * bestBary.x + e1TC * bestBary.y + aTC;

      // Return true, we hit something!
      return true;
   }

Note that this adds a lot of code duplication that should probably be cleaned up if the code were to remain in place even when a real fix is available.

You can now set the debug level from the console like

$AtlasInstance2::rayCollisionDebugLevel = 4;

My suggestion is to default ray collision to that level for the time being by setting the appropriate value in atlas/runtime/atlasInstance2.cpp.

Final note: I have seen further anomalies in Atlas mesh geometry, so I am currently not sure what part of Atlas's geometry processing really is to blame here.

#7 This is a great thread man. Good stuff.

#8 //EDIT: cleanup; support for blended terrain restored; fallback for PIX1.1
//NOTE: this is going to be superceded soon

Improvement: Optimization of texture I/O pipeline
TGEA Version: 1.7.0 final

Description

Atlas currently has a very costly texture I/O pipeline which shows mostly in it being very easy to outrun the pipeline with the camera. When this happens, you will see lower resolution textures on your terrain with higher resolution levels slowly popping in once the camera remains still.

One of the key issues here is the many steps and memory operations currently taken to bring a texture resource from offline to online state. One particular processing step where color values are being brought (swizzled) into a component order appropriate for the underlying graphics device, turns out to be particularly costly. The following changes move this step from the deserializer into the pixel shader.

However, this improvement works in such a way as to enable the change only if a) graphics hardware supports at least shader level 2.0 and b) unique texturing is used. Here is why:

a) Shader size restrictions in shader level 1.1 have practically already been reached with the current level 3 and level 4 Atlas shaders, so doing the swizzling in the shaders is not an option here.

b) Blended terrain will usually profit very little from this improvement as opacity and shadow maps usually will not amount to a lot of data. It is, however, easy to extend support to blended terrain; in this case, however, the blender shaders have to be adjusted the same way as the Atlas shaders.

Another noteworthy restriction of the current set of changes is that TGEA is implied to run on DirectX. Since TGEA at this point does not have support for other graphics APIs, I do not see this as a significant problem. In any way, to remove this restriction, it is only necessary to keep separate pixel shaders for each targeted API/device.

First results are encouraging. I have found it to not be possible anymore to outrun a single 4:1 texture map (512px tile size) with a flying camera at normal speeds. This should allow for greater texture resolutions, though the I/O pipeline still proves a severely limiting factor here and needs further improvement.

I have not seen any significant impact on frame rates.

Source Code

The following changes will make swizzling of texture maps optional and extend Atlas' pixel shader to automatically swap red and green values (this is specific to D3D), if desired.

First go to atlas/resource/atlasTexChunk.cpp@82 and replace the following code:

// For runtime we have to convert endianness.
      if(GFXDevice::getDeviceVector()->size())
      {
         if(bitmap[layer].bytesPerPixel == 4)
            bitmap[layer].swizzle(GFX->getDeviceSwizzle32());
         else if(bitmap[layer].bytesPerPixel == 3)
            bitmap[layer].swizzle(GFX->getDeviceSwizzle24());
      }

with

AssertFatal( dynamic_cast< AtlasResourceTexTOC* >( mOwningTOC ) != NULL,
		  "AtlasTexChunk::read - tex chunk that does not belong to tex TOC" );
	  AtlasResourceTexTOC* toc = ( AtlasResourceTexTOC* ) mOwningTOC;
	  if( toc->mSwizzleTextures && GFXDevice::getDeviceVector()->size() )
	  {
		  if(bitmap[layer].bytesPerPixel == 4)
			  bitmap[layer].swizzle(GFX->getDeviceSwizzle32());
		  else if(bitmap[layer].bytesPerPixel == 3)
			  bitmap[layer].swizzle(GFX->getDeviceSwizzle24());
	  }

Then go into atlas/resource/atlasResourceTexTOC.h and add the following public field to AtlasResourceTexTOC:

bool mSwizzleTextures;	///< If true, textures will be swizzled into device format when loaded.

Also add initialization to the constructor:

mSwizzleTextures = true;

Then add the following public field to ClipMap in clipmap/clipMap.h

bool mUseSwizzledShaders; ///< If true, an alternate set of shaders is loaded that does swizzling in the pixel shaders.

and add initialization to the constructor

mUseSwizzledShaders = false;

In clipmap/clipMap.cpp@152 replace

// Do shader lookup for 2,3,4 level shaders.
   for(S32 i=2; i<5; i++)
   {
      ShaderData *shader = NULL;
      const char *tmpBuff = avar("AtlasShader%d", i);
      if(!Sim::findObject(StringTable->insert(tmpBuff), shader))
      {
         Con::errorf("ClipMap::initClipStack - could not find shader '%s'!",tmpBuff);
         continue;
      }
      
      mShaders[i-1] = shader->shader;
   }

with

// Do shader lookup for 2,3,4 level shaders.
   const char* shaderNamePattern = "AtlasShader%d";
   if( mUseSwizzledShaders )
	   shaderNamePattern = "AtlasShader%dS";

   for(S32 i=2; i<5; i++)
   {
      ShaderData *shader = NULL;
      const char *tmpBuff = avar( shaderNamePattern, i );
      if(!Sim::findObject(StringTable->insert(tmpBuff), shader))
      {
         Con::errorf("ClipMap::initClipStack - could not find shader '%s'!",tmpBuff);
         continue;
      }
      
      mShaders[i-1] = shader->shader;
   }

After this, the last change to C++ source code will be responsible for deciding whether to enable swizzling in shaders.

First make sure you have this #include in place in atlas/runtime/atlasInstance2.cpp

#include "gfx/gfxDevice.h"

Then at line 123, add the following code

//EDIT: this previously stated that the code should be added at line 130 (as the last thing in the init block), but that leads to a bug where the topmost TOC level is set up incorrectly. It is important to add this code before the setTOC call.

bool useSwizzledShaders = ( GFX->getPixelShaderVersion() >= 2.0 );
			   mClipMap->mUseSwizzledShaders = useSwizzledShaders;
			   arttoc->mSwizzleTextures = !useSwizzledShaders;

This concludes the changes to be made to the engine.

Next, go into your game directory and in shaders/atlas/atlas.h@129 add the following

#ifdef SWIZZLE
   OUT.col = OUT.col.bgra;
#endif

In the same directory, copy each of the atlasSurfaceP?.hlsl shaders to a separate atlasSurfaceP?S.hlsl and add a #define SWIZZLE to each file.

Finally, to add the alternative set of shaders, go to scriptsAndAssets/client/scripts/shaders.cs@267 (again in your game directory) and add the following

new ShaderData( AtlasShader2S )
{
   DXVertexShaderFile   = "shaders/atlas/atlasSurfaceV2.hlsl";
   DXPixelShaderFile    = "shaders/atlas/atlasSurfaceP2S.hlsl";
   pixVersion = 2.0;
};
new ShaderData( AtlasShader3S )
{
   DXVertexShaderFile   = "shaders/atlas/atlasSurfaceV3.hlsl";
   DXPixelShaderFile    = "shaders/atlas/atlasSurfaceP3S.hlsl";
   pixVersion = 2.0;
};

new ShaderData( AtlasShader4S )
{
   DXVertexShaderFile   = "shaders/atlas/atlasSurfaceV4.hlsl";
   DXPixelShaderFile    = "shaders/atlas/atlasSurfaceP4S.hlsl";
   pixVersion = 2.0;
};

#9 Fix: Stubs remaining in "Loading" state
TGEA Version: 1.7.0 final

Description

This fix is more of a cosmetic change than a real fix. Even though the loading state of a stub is currently not correctly reset to "Unloaded" when chunk data is a DOA (dead on arrival), this won't cause any malfunctioning due to the way loading works in Atlas.

Source Code

In atlas/core/atlasResourceTOC.h@432, add the following:

stub->mState = StubType::Unloaded;

#10 Improvement: Object lighting sync'd to lighting kit lightmaps
TGEA Version: 1.7.0 final

Description

Jeff Faust uncovered a shortcoming (thread) where object lighting is not correctly adjusted in response to lightmaps when using Atlas. The following changes improve on this by adding respective lightmap queries to the lighting kit code.

Source Code

Add the following snippet to the public interface of AtlasInstance in atlas/runtime/atlasInstance2.h

AtlasFile* getLightMapFile()
   {
	   return mLightMapFile;
   }

Then, in lighting/synapseGaming/atlas/sgAtlas.cpp replace the existing sgAtlasSystem::getColorFromRayInfo method with the following code

bool sgAtlasSystem::getColorFromRayInfo(RayInfo collision, ColorF& result)
{
	if( !dynamic_cast< AtlasInstance* >( collision.object ) )
		return false;

	// Get to the lightmap file and retrieve the texture TOC.
	// Note that this code only works with the lightmaps produced by this module.
	// It does not work with lightmaps packaged with the main map (suppose there
	// is no need it does).

	AtlasInstance* atlas = ( AtlasInstance* ) collision.object;
	AtlasFile* lightMapFile = atlas->getLightMapFile();
	if( !lightMapFile )
		return false;
	AtlasResourceTexTOC* arttoc;
	if( !lightMapFile->getTocBySlot( 0, arttoc ) )
		return false;

	// Get some key figures.

	U32 treeDepth = arttoc->getTreeDepth();
	U32 chunkSize = arttoc->getTextureChunkSize();
	U32 lightmapSize = BIT( treeDepth - 1 ) * chunkSize;

	// Scale the texcoords to lightmap coords.

	Point2I lightmapCoords( collision.texCoord.x * F32( lightmapSize ),
		collision.texCoord.y * F32( lightmapSize ) );

	// Scale the lightmap coords into chunk coords.

	Point2I chunkXY( lightmapCoords.x / chunkSize, lightmapCoords.y / chunkSize );

	// Walk the TOC tree bottom up and stop at the first matching
	// piece of data that is available.

	for( int i = treeDepth - 1; i >= 0; -- i )
	{
		// Get the stub and, if its data is online, query its texture.

		AtlasResourceTexStub* stub = arttoc->getStub( i, chunkXY );
		if( stub->hasResource() )
		{
			// Data is there.  Grab the bitmap and read out the color information.

			if( !stub->mChunk->isBitmapTexFormat( stub->mChunk->mFormat ) )
				return false;
			
			GBitmap* bitmap = stub->mChunk->bitmap;

			U32 shift = treeDepth - i - 1;
			U32 x = ( lightmapCoords.x >> shift ) - chunkXY.x * chunkSize;
			U32 y = ( lightmapCoords.y >> shift ) - chunkXY.y * chunkSize;

			ColorI color;
			bitmap->getColor( x, y, color );

			result = ColorF( F32( color.red ) / 255.0 * 2.0, F32( color.green ) / 255 * 2.0,
				F32( color.blue ) / 255.0 * 2.0, 1.0 );

			return true;
		}
		else
		{
			// No data.

			//TODO: This is the place to request the data to be loaded.  However, as long
			//  as there isn't also a place where we kick out chunks no longer needed, this
			//  is not a reasonable thing to do.
			//arttoc->requestLoad( stub, AtlasTOC::NormalPriority,
			//	F32( treeDepth - i ) / F32( treeDepth + 1 ) );

			// When going up a level, we need to scale our texcoords
			// to the new level dimensions.

			chunkXY.x >>= 1;
			chunkXY.y >>= 1;
		}
	}

	return false;
}

Note: These changes only pertain to Atlas lighting information created by the SG Lighting Kit.

#11 Improvement: Prevent collision meshes from exceeding internal limits
TGEA Version: 1.7.0 final

Description

While meshes in Atlas are restricted to 65536 vertices per chunk, there is no (direct) limit on the number of triangles in a chunk (of course, the number of vertices will yield an upper limit). Collision meshes, however, that are derived from primary meshes use 16bit triangle indices and thus are restricted to 65536 triangles per chunk.

While exceeding this limit already indicates a problem with your geometry tree that calls for a deeper tree, there is currently no feedback about this in Atlas and violations will indirectly resurface later as raycasting failures.

The following change adds code to the old CHU generator to indicate failure in case the triangle limit is exceeded with collision meshes.

Source Code

Add the following to atlas/editor/atlasOldMesher.cpp@312:

// Limit the triangle count to 16bits.  While primary meshes support more
		 // than that, collision meshes don't.  If we don't catch that here, we'll
		 // see raycasting issues later in Atlas.
		 AssertISV( binList.size() <= 65536,
			 "AtlasOldMesher::writeCollision - too many triangles! (>65536)  Try again with a deeper tree" );