TGEA 1.7.1 Many memory leak
by Bartholomew IU · in Torque Game Engine Advanced · 09/25/2008 (11:27 pm) · 4 replies
I use the Stronghold example to test the memory leak. My source code is a fresh install with some other patches found in this forum. So you may not able to use my patch file, sorry.
I first try this: www.garagegames.com/mg/forums/result.thread.php?qt=76462
to make the Torque memory manager (TMM) work again. It is hard to use TMM to check the memory leak of the whole program. Then I try the memory leak detection from Microsoft:
msdn.microsoft.com/en-us/library/e5ewb1h3(VS.80).aspx
This memory leak detection works correctly in multi thread and COM program. There is another detection tools with stack trace:
www.codeproject.com/KB/applications/visualleakdetector.aspx
, but this one does not work correctly (has false alarm) in multi thread program.
To use the Microsoft detection method, I need to add some lines to all files that call malloc. TGEA does not use precompiled header, so I need to add these lines:
Then I need to call this in the main function:
www.filesavr.com/memorycheck1_1
After compiled the program in debug mode and run in MSVC, you can see that there are many memory leak. Some of them have the file name and line number. For those without file name and line number, it may be the leak caused by "new". The detection code for new is rather complicated. I need to disable TMM and add these in torqueConfig.h:
The patch for this part is here:
www.filesavr.com/memorycheck2
Doing this by hand takes time. So I write a small program to add the header. This program is compiled using wxWidgets 2.8.7. To use it, replace the console.cpp in console example and compile it using ascii debug build (haven't tried other build yet), and run it with the source code directory of TGEA as the first parameter. Here is my small program:
www.filesavr.com/console_1
Well, there are still some leaks without file name and line number. I guess it is caused by the "new" statement in the header file. Hope that someone will move the functions with "new" into cpp file.
To enable the break point on the memory allocation number, try this:
msdn.microsoft.com/en-us/library/w2fhc9a3(VS.80).aspx
Hope that the GG people can kill all the memory leak and release a new version soon.
I first try this: www.garagegames.com/mg/forums/result.thread.php?qt=76462
to make the Torque memory manager (TMM) work again. It is hard to use TMM to check the memory leak of the whole program. Then I try the memory leak detection from Microsoft:
msdn.microsoft.com/en-us/library/e5ewb1h3(VS.80).aspx
This memory leak detection works correctly in multi thread and COM program. There is another detection tools with stack trace:
www.codeproject.com/KB/applications/visualleakdetector.aspx
, but this one does not work correctly (has false alarm) in multi thread program.
To use the Microsoft detection method, I need to add some lines to all files that call malloc. TGEA does not use precompiled header, so I need to add these lines:
#define _CRTDBG_MAP_ALLOC #include <stdlib.h> #include <crtdbg.h>in a common header. I think torqueConfig.h is a good place to add the lines. These lines should be surrounded by some marco so that it will only be compiled in MSVC, but I don't care it now.
Then I need to call this in the main function:
_CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );When I compile the program, the compiler gives me many errors. This is caused by the function name "free" and "realloc". The list of prohibited words is in the crtdbg.h under the _CRTDBG_MAP_ALLOC macro. Then I need to change the function name or surrounded them by a special compiler directive:
#pragma push_macro("realloc")
#undef realloc
......
#pragma pop_macro("realloc")I hope the GG people can change the function name instead of using this ugly directive. The patch for this part is here:www.filesavr.com/memorycheck1_1
After compiled the program in debug mode and run in MSVC, you can see that there are many memory leak. Some of them have the file name and line number. For those without file name and line number, it may be the leak caused by "new". The detection code for new is rather complicated. I need to disable TMM and add these in torqueConfig.h:
#ifdef _DEBUG
#define DEBUG_NEW new( _NORMAL_BLOCK, __FILE__, __LINE__)
// Replace _NORMAL_BLOCK with _CLIENT_BLOCK if you want the
//allocations to be of _CLIENT_BLOCK type
#else
#define DEBUG_NEW
#endif // _DEBUGThen I need to add these in every cpp files after all the "include" statements (If not after all include statements, it will cause other errors, for example, errors in STL header if you use it.):#ifdef _DEBUG #define new DEBUG_NEW #endifInstead of adding the above lines directly, I put the lines into a header file "memory_leak_check.h" and include this header in the cpp files.
The patch for this part is here:
www.filesavr.com/memorycheck2
Doing this by hand takes time. So I write a small program to add the header. This program is compiled using wxWidgets 2.8.7. To use it, replace the console.cpp in console example and compile it using ascii debug build (haven't tried other build yet), and run it with the source code directory of TGEA as the first parameter. Here is my small program:
www.filesavr.com/console_1
Well, there are still some leaks without file name and line number. I guess it is caused by the "new" statement in the header file. Hope that someone will move the functions with "new" into cpp file.
To enable the break point on the memory allocation number, try this:
msdn.microsoft.com/en-us/library/w2fhc9a3(VS.80).aspx
Hope that the GG people can kill all the memory leak and release a new version soon.
#2
09/25/2008 (11:38 pm)
Patch 1 part 2:Index: engine/source/gfx/gfxStateFrame.cpp
===================================================================
--- engine/source/gfx/gfxStateFrame.cpp (revision 2)
+++ engine/source/gfx/gfxStateFrame.cpp (working copy)
@@ -40,7 +40,10 @@
// Con::printf(" setting state %d=%d", walkG->state, walkG->oldVal);
GFX->trackRenderState(walkG->state, walkG->oldVal);
walkG = walkG->next;
+ #pragma push_macro("free")
+ #undef free
mGlobalChunker.free(del);
+ #pragma pop_macro("free")
}
Fragment *walk = mTexture;
@@ -50,7 +53,10 @@
// Con::printf(" setting mTexture %d state %d=%d", walk->stage, walk->state, walk->oldVal);
GFX->trackTextureStageState(walk->stage, walk->state, walk->oldVal);
walk = (Fragment*)walk->next;
+ #pragma push_macro("free")
+ #undef free
mFragmentChunker.free(del);
+ #pragma pop_macro("free")
}
walk = mSampler;
@@ -60,7 +66,10 @@
// Con::printf(" setting mSampler %d state %d=%d", walk->stage, walk->state, walk->oldVal);
GFX->trackSamplerState(walk->stage, walk->state, walk->oldVal);
walk = (Fragment*)walk->next;
+ #pragma push_macro("free")
+ #undef free
mFragmentChunker.free(del);
+ #pragma pop_macro("free")
}
mEnable = true;
Index: engine/source/gfx/gfxTextureHandle.h
===================================================================
--- engine/source/gfx/gfxTextureHandle.h (revision 2)
+++ engine/source/gfx/gfxTextureHandle.h (working copy)
@@ -42,7 +42,11 @@
U32 getDepth() { return getPointer() ? getPointer()->getDepth() : 0; }
void refresh();
+
+ #pragma push_macro("free")
+ #undef free
void free(); ///< Release any resources attached to this object.
+ #pragma pop_macro("free")
GFXLockedRect *lock( U32 mipLevel = 0, RectI *inRect = NULL )
{
Index: engine/source/platform/platformMemory.cpp
===================================================================
--- engine/source/platform/platformMemory.cpp (revision 2)
+++ engine/source/platform/platformMemory.cpp (working copy)
@@ -1166,7 +1166,10 @@
#endif
#if !defined(TORQUE_DISABLE_MEMORY_MANAGER)
+#pragma push_macro("free")
+#undef free
static void free(void* mem, bool array)
+#pragma pop_macro("free")
{
// validate();
@@ -1248,11 +1251,17 @@
#endif
#if !defined(TORQUE_DISABLE_MEMORY_MANAGER)
+#pragma push_macro("realloc")
+#undef realloc
static void* realloc(void* mem, dsize_t size, const char* fileName, const U32 line)
+#pragma pop_macro("realloc")
{
//validate();
if (!size) {
+ #pragma push_macro("free")
+ #undef free
free(mem, false);
+ #pragma pop_macro("free")
return NULL;
}
if(!mem)
@@ -1340,7 +1349,10 @@
#endif
void* ret = alloc(size, false, fileName, line);
dMemcpy(ret, mem, oldSize);
+ #pragma push_macro("free")
+ #undef free
free(mem, false);
+ #pragma pop_macro("free")
PROFILE_END();
// Re-enable the 'Reallocated' flag so that this allocation can be ignored by
@@ -1449,12 +1461,18 @@
void FN_CDECL operator delete(void* mem)
{
+ #pragma push_macro("free")
+ #undef free
Memory::free(mem, false);
+ #pragma pop_macro("free")
}
void FN_CDECL operator delete[](void* mem)
{
+ #pragma push_macro("free")
+ #undef free
Memory::free(mem, true);
+ #pragma pop_macro("free")
}
void* dMalloc_r(dsize_t in_size, const char* fileName, const dsize_t line)
@@ -1464,12 +1482,18 @@
void dFree(void* in_pFree)
{
+ #pragma push_macro("free")
+ #undef free
Memory::free(in_pFree, false);
+ #pragma pop_macro("free")
}
void* dRealloc_r(void* in_pResize, dsize_t in_size, const char* fileName, const dsize_t line)
{
+ #pragma push_macro("realloc")
+ #undef realloc
return Memory::realloc(in_pResize, in_size, fileName, line);
+ #pragma pop_macro("realloc")
}
#else
Index: engine/source/sceneGraph/lightAllocator.cpp
===================================================================
--- engine/source/sceneGraph/lightAllocator.cpp (revision 2)
+++ engine/source/sceneGraph/lightAllocator.cpp (working copy)
@@ -20,7 +20,10 @@
void LightAllocator::clear()
{
+ #pragma push_macro("free")
+ #undef free
free();
+ #pragma pop_macro("free")
for ( S32 i = 0; i < mFreeLights.size(); i++ )
delete mFreeLights[i];
Index: engine/source/sceneGraph/lightAllocator.h
===================================================================
--- engine/source/sceneGraph/lightAllocator.h (revision 2)
+++ engine/source/sceneGraph/lightAllocator.h (working copy)
@@ -37,7 +37,10 @@
LightInfo* alloc( LightManager* lm );
/// Moves all the lights into the free list.
+ #pragma push_macro("free")
+ #undef free
void free();
+ #pragma pop_macro("free")
protected:
Vector<LightInfo*> mFreeLights;
Index: engine/source/ts/tsShape.h
===================================================================
--- engine/source/ts/tsShape.h (revision 2)
+++ engine/source/ts/tsShape.h (working copy)
@@ -523,7 +523,10 @@
TSMaterialList();
TSMaterialList(const TSMaterialList*);
~TSMaterialList();
+#pragma push_macro("free")
+ #undef free
void free();
+#pragma pop_macro("free")
void load(U32 index, const char* path = 0);
bool load(TextureHandleType type, const char* path = 0,bool clampToEdge = false) { return Parent::load(type,path,clampToEdge); }
Index: engine/source/util/journal/journal.cpp
===================================================================
--- engine/source/util/journal/journal.cpp (revision 2)
+++ engine/source/util/journal/journal.cpp (working copy)
@@ -52,7 +52,10 @@
if((*itr)->match(ptr, method))
{
// Unlink and break.
+ #pragma push_macro("free")
+ #undef free
idPool().free((*itr)->id);
+ #pragma pop_macro("free")
*itr = (*itr)->next;
return;
}
Index: GameExamples/Stronghold/source/main.cpp
===================================================================
--- GameExamples/Stronghold/source/main.cpp (revision 2)
+++ GameExamples/Stronghold/source/main.cpp (working copy)
@@ -2,6 +2,9 @@
#include "app/mainLoop.h"
#include "T3D/gameFunctions.h"
+#ifdef _DEBUG
+ #define new DEBUG_NEW
+#endif
// Entry point for your game.
//
@@ -10,6 +13,11 @@
// will need to merge against future changes to the SML code if you do this.
S32 TorqueMain(S32 argc, const char **argv)
{
+ _CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
+
+ char *a = new char[10];
+ void *b = malloc(10);
+
// Some handy debugging code:
// if (argc == 1) {
// static const char* argvFake[] = { "dtest.exe", "-jload", "test.jrn" };
Index: GameExamples/Stronghold/source/torqueConfig.h
===================================================================
--- GameExamples/Stronghold/source/torqueConfig.h (revision 3)
+++ GameExamples/Stronghold/source/torqueConfig.h (working copy)
@@ -6,6 +6,18 @@
#ifndef _TORQUECONFIG_H_
#define _TORQUECONFIG_H_
+#define _CRTDBG_MAP_ALLOC
+#include <stdlib.h>
+#include <crtdbg.h>
+
+#ifdef _DEBUG
+ #define DEBUG_NEW new( _NORMAL_BLOCK, __FILE__, __LINE__)
+ // Replace _NORMAL_BLOCK with _CLIENT_BLOCK if you want the
+ //allocations to be of _CLIENT_BLOCK type
+#else
+ #define DEBUG_NEW
+#endif // _DEBUG
+
//-----------------------------------------------------------------------------
//Hi, and welcome to the Torque Config file.
//
@@ -41,7 +53,7 @@
#define TORQUE_MULTITHREAD
/// Define me if you want to disable Torque memory manager.
-//#define TORQUE_DISABLE_MEMORY_MANAGER
+#define TORQUE_DISABLE_MEMORY_MANAGER
/// Define me if you don't want Torque to compile dso's
#define TORQUE_NO_DSO_GENERATION
#3
However, my small program has a unknown memory leak :P
09/25/2008 (11:41 pm)
Patch 2 is very large, so I only post my small program.However, my small program has a unknown memory leak :P
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
#include "wx/wx.h"
#include <wx/arrstr.h>
#include <wx/dir.h>
#include <wx/textfile.h>
#ifdef _DEBUG
#define DEBUG_NEW new( _NORMAL_BLOCK, __FILE__, __LINE__)
// Replace _NORMAL_BLOCK with _CLIENT_BLOCK if you want the
//allocations to be of _CLIENT_BLOCK type
#else
#define DEBUG_NEW
#endif // _DEBUG
#ifdef _DEBUG
#define new DEBUG_NEW
#endif
#define addThisString "#include \"memory_leak_check.h\""
void addHeader(wxString &file){
wxTextFile textFile(file);
textFile.Open();
int lastIncludeOccur = 0;
int lineCount = textFile.GetLineCount();
for(int i = 0; i < lineCount; i ++){
wxString line = textFile.GetLine(i);
line.Trim(false);
if (line.StartsWith("#include")){
lastIncludeOccur = i;
}
}
wxLogMessage("Last: %d", lastIncludeOccur);
textFile.InsertLine(wxString::Format("%s // Debug memory leak checking for new", addThisString), lastIncludeOccur+1);
textFile.InsertLine("", lastIncludeOccur+1);
textFile.Write();
}
int main(int argc, char **argv)
{
if (argc < 2){
wxLogMessage("Parameter 1 is the target directory.\nAll the *.cpp in the subfolder will be added this line after all include: %s", addThisString);
return 0;
}
wxString targetPath = argv[1];
{
wxDir dir(targetPath);
if ( !dir.IsOpened() )
{
// deal with the error here - wxDir would already log an error message
// explaining the exact reason of the failure
wxLogMessage("The dir: \"%s\" cannot be opened");
return 0;
}
}
wxArrayString files;
int noOfFiles = wxDir::GetAllFiles(targetPath, &files, "*.cpp", wxDIR_DEFAULT);
for(int i = 0; i < noOfFiles; i ++){
wxString &file = files[i];
wxLogMessage("%s", file.c_str());
addHeader(file);
}
return 0;
}
#4
DXDiagNVUtil.cpp and getdxver.cpp
09/26/2008 (1:35 am)
I forget one thing. "memory_leak_check.h" should not be added in these two files:DXDiagNVUtil.cpp and getdxver.cpp
Torque Owner Bartholomew IU
Patch 1:
Index: engine/source/collision/convex.cpp =================================================================== --- engine/source/collision/convex.cpp (revision 2) +++ engine/source/collision/convex.cpp (working copy) @@ -36,10 +36,13 @@ CollisionState::~CollisionState() { + #pragma push_macro("free") + #undef free if (mLista) mLista->free(); if (mListb) mListb->free(); + #pragma pop_macro("free") } void CollisionState::swap() @@ -245,7 +248,10 @@ return constructInPlace((CollisionStateList*)sChunker.alloc(sizeof(CollisionStateList))); } +#pragma push_macro("free") +#undef free void CollisionStateList::free() +#pragma pop_macro("free") { unlink(); linkAfter(&sFreeList); @@ -297,7 +303,10 @@ return constructInPlace((CollisionWorkingList*)sChunker.alloc(sizeof(CollisionWorkingList))); } +#pragma push_macro("free") +#undef free void CollisionWorkingList::free() +#pragma pop_macro("free") { unlink(); wLinkAfter(&sFreeList); @@ -327,6 +336,8 @@ while (mList.mNext != &mList) delete mList.mNext->mState; + #pragma push_macro("free") + #undef free // Free up working list while (mWorking.wLink.mNext != &mWorking) mWorking.wLink.mNext->free(); @@ -334,6 +345,7 @@ // Free up references while (mReference.rLink.mNext != &mReference) mReference.rLink.mNext->free(); + #pragma pop_macro("free") } @@ -452,7 +464,10 @@ if ((!box.isOverlapped(itr->mConvex->getBoundingBox())) || (!itr->mConvex->getObject()->isCollisionEnabled())) { CollisionWorkingList* cl = itr; itr = itr->wLink.mPrev; + #pragma push_macro("free") + #undef free cl->free(); + #pragma pop_macro("free") } } Index: engine/source/collision/convex.h =================================================================== --- engine/source/collision/convex.h (revision 2) +++ engine/source/collision/convex.h (working copy) @@ -108,7 +108,10 @@ bool isEmpty() { return mNext == this; } static CollisionStateList* alloc(); + #pragma push_macro("free") + #undef free void free(); + #pragma pop_macro("free") }; @@ -133,7 +136,10 @@ CollisionWorkingList(); static CollisionWorkingList* alloc(); + #pragma push_macro("free") + #undef free void free(); + #pragma pop_macro("free") }; Index: engine/source/core/dataChunker.h =================================================================== --- engine/source/core/dataChunker.h (revision 2) +++ engine/source/core/dataChunker.h (working copy) @@ -106,6 +106,8 @@ return ret; } + #pragma push_macro("free") + #undef free void free(T* elem) { numAllocated--; @@ -119,6 +121,7 @@ freeListHead = NULL; } } + #pragma pop_macro("free") // Allow people to free all their memory if they want. void freeBlocks() Index: engine/source/core/idGenerator.h =================================================================== --- engine/source/core/idGenerator.h (revision 2) +++ engine/source/core/idGenerator.h (working copy) @@ -60,6 +60,8 @@ return mNextId++; } + #pragma push_macro("free") + #undef free void free(U32 id) { AssertFatal(id >= mIdBlockBase, "IdGenerator::alloc: invalid id, id does not belong to this IdGenerator.") @@ -71,6 +73,7 @@ else mPool.push_back(id); } + #pragma pop_macro("free") U32 numIdsUsed() { Index: engine/source/core/llist.h =================================================================== --- engine/source/core/llist.h (revision 2) +++ engine/source/core/llist.h (working copy) @@ -254,7 +254,10 @@ //--------------------------------------------------------------------------------------- // Unlink item from list and destroy it //--------------------------------------------------------------------------------------- + #pragma push_macro("free") + #undef free void free(T *entry) + #pragma pop_macro("free") { unlink(entry); delete entry; @@ -263,7 +266,10 @@ //--------------------------------------------------------------------------------------- // Unlink and destroy all list items //--------------------------------------------------------------------------------------- + #pragma push_macro("free") + #undef free void free(void) + #pragma pop_macro("free") { LListNode<T> *node = NULL;