Memory Leaks @ Vanilla Torque

#21 Thanks, Orion!

#22 LEAK: console/consoleObject.cc line 150
classTable[group][type] = new AbstractClassRep*[NetClassCount[group][type]];

SEVERITY: once every application startup and shutdown

FIX:
1. add to console/consoleObject.h line 275

// + KIM 081106 LEAK_FIX
   static void uninitialize(); // Called from Con::shutdown once on shutdown
   // - KIM 081106 LEAK_FIX

2. add to console/consoleObject.cc line 171

// + KIM 081106 LEAK_FIX
void AbstractClassRep::uninitialize()
{
	for (U32 group = 0; group < NetClassGroupsCount; group++)
	{
		for(U32 type = 0; type < NetClassTypesCount; type++)
		{
			if ( classTable[group][type] )
			{
				delete[] classTable[group][type];
			}
		}
	}
}
// - KIM 081106 LEAK_FIX

3. add to console/console.cc line 263

void shutdown()
{
   AssertFatal(active == true, "Con::shutdown should only be called once.");
   active = false;

   consoleLogFile.close();
   Namespace::shutdown();

   Compiler::freeConsoleParserList(); // + KIM 081104 LEAK_FIX
   AbstractClassRep::uninitialize(); // + KIM 081106 LEAK_FIX
}

#23 LEAK:
sim/netStringTable.cc line 19 - table = (Entry *) dMalloc(sizeof(Entry) * InitialSize);
sim/netStringTable.cc line 68 - table = (Entry *) dRealloc(table, newSize * sizeof(Entry));

FIX:
add to sim/netStringTable.cc line 34

NetStringTable::~NetStringTable()
{
	// + KIM 081106 b LEAK_FIX
	if ( table )
		dFree( table );
	// - KIM 081106 b LEAK_FIX
   delete allocator;
}

#24 LEAK:
core/chunkFile.cc line 237 - FourCCToAcr * fcta = new FourCCToAcr();

FIX:
1. add to core/chunkFile.h line 60

// + KIM 081106 c LEAK_FIX
   static void uninitChunkMappings();
   // - KIM 081106 c LEAK_FIX

2. add to core/chunkFile.cc line

// + KIM 081106 c LEAK_FIX
void SimChunk::uninitChunkMappings()
{
	Con::printf("Uninitializing chunk mappings...");

	for( U32 i = 0; i < smFourCCList.size(); i++ )
	{
		if ( smFourCCList[i] )
			delete smFourCCList[i];
	}
}
// - KIM 081106 c LEAK_FIX line 251

3. add to game/main.cc line 375 (on last line of function void shutdownGame())

// + KIM 081106 c LEAK_FIX
   SimChunk::uninitChunkMappings();
   // - KIM 081106 c LEAK_FIX

#25 LEAK
dgl/gTexManager.cc line 1641 - ret = new ChunkedTextureObject;
function ChunkedTextureManager::registerTexture(const char *textureName, GBitmap *data, bool keep)

SEVERITY: fatal

FIX:
add to dgl/gTexManager.cc line 1676

void ChunkedTextureManager::freeTexture(ChunkedTextureObject *to)
{
   // remove it from the linked list

   for(ChunkedTextureObject **walk = &gChunkedTextureList; *walk; walk = &((*walk)->next))
   {
      if(*walk == to)
      {
         *walk = to->next;
         delete[] to->textureHandles;
         delete to->bitmap;
		 // + KIM 081106 d LEAK_FIX
		 delete to;
		 // - KIM 081106 d LEAK_FIX
         return;
      }
   }
}

#26 Just so you know there are people really appreciating you taking the time to dig this stuff out and post it. Just want to say thanks. Kind of surprising with torque being out all these years that this had not been done yet.

keep up the good work!

#27 Hey Bullitt,
Just want to say awesome freakin work man!! I was aware of this memory leak last year but had no idea where to start fixing it. It blows my mind that GG hasn't fixed this. Anyways, I've gone through and made all the changes but for some reason I keep getting this error message:

error C2660: 'Compiler::addConsoleParser' :function does not take 8 arguments.

When I click on the error it comes to these two lines in console.cc line 234 ish

// Set up the parser(s)
CON_ADD_PARSER(CMD,   "cs",   true);   // TorqueScript
CON_ADD_PARSER(BAS,   "bas",  false);  // TorqueBasic

I did make the change in consoleParser.h line 79 ish to:

#define CON_ADD_PARSER(prefix, ext, def) \
	Compiler::addConsoleParser(ext, prefix##GetCurrentFile, prefix##GetCurrentLine, prefix##parse, \
		prefix##restart, prefix##SetScanBuffer, prefix##shutdown, def)

Is there anything else that might have caused this error?
Thx.

#28 Ah, found it. I did not add the extra parameter in the addConsoleParser, in the header file.

But now when I run the game it instantly crashes. In debug mode it stops at platformMemory.cc line 484ish where it sais this:

FreeHeader *prev = hdr->prevQueue;
   FreeHeader *next = hdr->nextQueue;

   if(prev)
      prev->nextQueue = next; // PROGRAM BREAKS HERE.......?
   else
      hdr->treeNode->queueHead = next;

Thx.

#29 Hi all,
thanks for all your supports! Hope I to be able to get through the leaks in core/dataChunker.cc though, it happens so often and it leaks 16K each. I've been tracing lots of time, and it seems fine / disposed properly, but the CRT detector still report those leaks...

@DALO
I've just checked my built on vanilla and my game. It did not happen on both, with and without Torque's memory manager. But I am not using Torque's memory manager for various reasons including this one: http://www.garagegames.com/index.php?sec=mg&mod=resource&page=view&qid=6190 , furthermore, the memory leaks report generated by Torque's memory manager are sometimes irrelevant, pointing elsewhere or reporting as leak to otherwise safe codes.

Please note, however, I did not apply the patch in that resource. I just define TORQUE_DISABLE_MEMORY_MANAGER on the preprocessor in Torque Demo's project properties.

#30 Ah! I got it! Those reported leaks comes from global static objects with file linkage, whose destructors were only called after main() returns on crt0.c, whereas I dump my memory leak reports just before main() returns. That is probably why I still get all those leak reports even after cleaning!

However, I think I still need to check more thoroughly as there are still 3K lines of leaks reported. Who knows some 'real' leaks still slip through..

At least can rest assured some reported leaks are safe codes.

#31 @Bullitt

Thanks,by adding TORQUE_DISABLE_MEMORY_MANAGER, that problem seems solved,but...
rarely it crashes near

void PEngine::updateSingleParticle(Particle* particle, ParticleEmitter &emitter, const U32 ms)
{
   AssertFatal(particle != NULL, "PEngine::updateSingleParticle: Error, must have a particle to process in this function");
   AssertFatal(ms != 0, "PEngine::updateSingleParticle: error, no time to update?");

   F32 t = F32(ms) / 1000.0f;

   Point3F a = particle->acc;
   a -= particle->vel        * particle->dataBlock->dragCoefficient;  <----- Crash,daraBlock was invalid.

I'm still wondering what situation would cause this.

#32 Hey Bullitt,
Would it be possible for you to post all the files changed as a resource or a link to download? I have made all the changes and double checked all my code. It still instantly crashes at startup. So I decided to go start all over and go 1 step at a time. I started off with the steps 1-16 and had no problems starting the game, however when I would stop and restart missions, nothing changed in terms of memory, it still gradually increased. I eventually started adding all the other posts here and there and would recompile and test each time until I got to the post LEAK: console/consoleObject.cc line 150 and the next post. Then I started getting bizarre errors with files in the dgl/ folder??? Also it would tell me I had errors in locations that don't exist on my drive like c:\cvs\tge\engine\...... I back tracked those steps and somehow it just started turning into a big mess of errors. I cleaned the solution and project files and still had some funky issues. So I just reverted back to the default engine files.....sadly......I don't want these leaks to creep up on me. I'd say I'm almost advanced, but not quite there when it comes to compiling and debugging but these errors were just unexplainable?
So again, could you pleeeeease post the files. Also what does vanilla mean?
Thx.

#33 @DALO:

The majority of these fixes [didn't look at all of them in detail] are for static data Torque is allocating. This is automatically freed when you exit an application. This is why you don't see a change in the memory profile while running Torque after applying these fixes.

There are two schools of thought on this:

1) The static data should be freed because it makes it easier to find 'real' leaks.
2) Forget about it because they aren't really leaks - the memory is cleaned up when the program quits, so why add more cycles at exit to clean up?

Traditionally Torque developers have chosen the second viewpoint which is why they haven't been 'fixed'.

#34 And 'vanilla' means code with no modifications.

#35 @DALO:
sorry, I don't have accessible public server. If you want, I can send the whole vanilla torque with LEAK_FIX signature to you (~3Mb) via e-mail or you could recommend any public free file server.

Andy is right, majority of these fixes are for global static objects with file linkage, which is only got cleaned up when the program quits (during shutdown calls). The purpose is to minimize the leak report size to easier spot the real leaks. You can easily spot these "helper fixes": they are usually called during game shutdown or after your game loop exits. You can safely ignore these though, but I'd prefer it there for cleaner execution footprint. The console parser is one of the helper fix that just happen once per execution. For the severe leaks (eg. from local variables), I've marked those fixes with higher SEVERITY warns.

For the misleading error location (C:\cvs\tge...) is because there are pragma #line defined within CMDgram.cc, CMDscan.cc, BASscan.cc etc.., you can comment out those #lines for easier debugging.

About the memory footprint that keep on increasing without ever reaching back its initial value when you continuously start and end mission..., the most memory consumption operation so far I noticed is in Torque's logging process. To quick kill the log, go to console\console.cc and comment line 507 - 509

//            entry.mString = (const char *)consoleLogChunker.alloc(dStrlen(pos) + 1);
//            dStrcpy(const_cast<char*>(entry.mString), pos);
//            consoleLog.push_back(entry);

You will notice the memory footprint going down to the level of after first mission loading if the same mission is loaded repeatedly.

#36 When I first read this forum post Bullitt describe an issue that I've been aware of for quite sometime now. To me this is a big deal that every time you quite a mission and start again the memory keeps climbing up crashing at a high memory number of approximately 2gigs. I must have misinterpreted the overall code part thinking it was a fix to that issue......my bad. Thanks for the replies and the code btw.

#37 Yeah, I started all this work because of that issue as well, suspecting the behavior as leak... going on to dump 3000+ lines of memory leak log and clearing them one by one.

But finally I found out that the issue is because the console print that is never been cleared. The quick solution I posted on Nov 11, 2008 10:07 do the trick for me, as I am going to disable the console.

That's all the hard hit blindly around the bushes, but I'm glad that I found it out at last and now I've got a 'clean' torque that reports 0 memory leaks.

#38 Hey Bullitt,
are you proposing just not keeping the console history around in memory,
or disabling the logging altogether ?
if the former, if you feel like posting how you do it that would same us some time.

yeah, having a codebase without leaks is a great feeling.

#39 Well, that's good to see 0 memory leaks. Thanks for all the hard work you have put into this, much appreciated. Oh and yes, I'd love an email with the changes. Thanks again.

#40 Hi Bullitt, and thanks for your hard work first !

Does this thread list all your modifications ?

Because I will integrate it in my TGE : it's always better to have NO memory leak, them being hazardous or not. It's like the warning during compilation : you shouldn't have any at the end !

Nicolas Buquet
www.buquet-net.com/cv/