Game Development Community

dev|Pro Game Development Curriculum

STL

by Dean Calver · 04/03/2005 (9:47 am) · 30 comments

2009 Update!!! TGEA and I expect Torque3D work out the box with STL now, so this nastiness should be a thing of the past thankfully :)

<a href="/static/pg/resource/0.stl_fix.h">Download Code File</a>

STL fix
The STL is a massive collection of classes that all use a pattern called allocators for all memory allocation stuff. The particular type of macros used by the Torque engine memory managment system cause most (certainly the windows and linux standard versions) STL allocators to not compile.
One 'solution' is to disable the torque memory manager completely but the torque memory manager does lots of good things, especially in the realms of debugging. It will save your arse at least once before you ship your game, so disabling it is BAD.
So a better fix is while keeping the existing torque memory manager replace the provided STL allocators with ones that work 'nicely' with torque.

This proves to be a path of darkness and true system level hackery as in there infinite wisdom, the C++ standards commitee have proved no way of replacing the default allocator (you can provide optionally different allocators, which is useful for pooled allocation but not replace the default allocator).
The problem is that each platform and compiler use different implementation of std::allocator so I needed to figure out how to get rid of each one in turn.

This major addition is stl_fix.h, its needs including before any stl header is included and then is just works... As new platform, compilers etc. come in stl_fix.h will need updating but I think I've covered the current windows side of things and probably Linux. Mac I don't know as I don't have access to one.

Well it almost "just works", you see the macro form of new that torque uses still upsets some of the more obscure parts of some implementations of STL. The only solution is to patch torque, the actual patch is no more than a few lines and loses no functionality.
I tried for a long time to come up with a solution that didn't require a torque patch but in the end I had to fallback to a patch.

How to get it worked

Thomus Lund's excellent GameSWG resource needed to use stl_fix, so helped sort out a few bugs and he also wrote up the instructions for how to use it. As Thomus's explaniation is so good, I've basically copied his instructions into this seperate resource.

Attached to this resource is stl_fix.h this should be copied to your engine/core directory. Include this before any stl headers.

Open up platform/platform.h
Find this in Line 396
extern void* FN_CDECL operator new(dsize_t size, const char*, const U32);
extern void* FN_CDECL operator new[](dsize_t size, const char*, const U32);
extern void  FN_CDECL operator delete(void* ptr);
extern void  FN_CDECL operator delete[](void* ptr);
#define new new(__FILE__, __LINE__)
and change to this
extern void* FN_CDECL operator new(dsize_t size);
extern void* FN_CDECL operator new[](dsize_t size);
extern void  FN_CDECL operator delete(void* ptr);
extern void  FN_CDECL operator delete[](void* ptr);
extern void dPopMemManFileLine( const char*&, U32& );
extern int dPushMemManFileLine( const char*, const U32 );
#define new dPushMemManFileLine( __FILE__, __LINE__ ) ? 0 : new

Open the file platform/platformMemory.cc

Find this in line 1080
#if !defined(TORQUE_DISABLE_MEMORY_MANAGER)

// Manage our own memory, add overloaded memory operators and functions

void* FN_CDECL operator new(dsize_t size, const char* fileName, const U32 line)
{
   return Memory::alloc(size, false, fileName, line);
}

void* FN_CDECL operator new[](dsize_t size, const char* fileName, const U32 line)
{
   return Memory::alloc(size, true, fileName, line);
}

void* FN_CDECL operator new(dsize_t size)
{
   return Memory::alloc(size, false, NULL, 0);
}

void* FN_CDECL operator new[](dsize_t size)
{
   return Memory::alloc(size, true, NULL, 0);
}
and change to this
#if !defined(TORQUE_DISABLE_MEMORY_MANAGER)

static const int NUM_NEW_STACK_SIZE = 256;
static struct { const char* filename; U32 line;}

g_NewStackMemDebug[NUM_NEW_STACK_SIZE];
static int g_CurStack = -1;

int dPushMemManFileLine( const char* filename, U32 line )
{
   if(g_CurStack >= NUM_NEW_STACK_SIZE )
       return 0;
 
    g_CurStack++;
    g_NewStackMemDebug[g_CurStack].filename = filename;
    g_NewStackMemDebug[g_CurStack].line = line;
 
    return 0; // needed for the new passthrough trick

}

void dPopMemManFileLine( const char*& filename, U32& line )
{
   if( g_CurStack < 0 )
    {
       filename = 0;
       line = 0;
       return;
    } else
    {
       filename = g_NewStackMemDebug[g_CurStack].filename;
       line = g_NewStackMemDebug[g_CurStack].line;
       g_CurStack--;
    }
}

// Manage our own memory, add overloaded memory operators and functions

void* FN_CDECL operator new(dsize_t size)
{
   const char* fileName = 0;
    U32 line;
    dPopMemManFileLine( fileName, line );
     return Memory::alloc(size, false, fileName, line);
}

void* FN_CDECL operator new[](dsize_t size)
{
   const char* fileName = 0;
    U32 line;
    dPopMemManFileLine( fileName, line );
    return Memory::alloc(size, true, fileName, line);
}

open platformWin32/winMemory.cc
Find this at Line 41
void* FN_CDECL operator new(dsize_t, void* ptr)
{
   return (ptr);
}
and change to this
void* FN_CDECL operator new(dsize_t, void* ptr)
{
    const char* fileName = 0;
    U32 line;
    // dummy does nothing except keep the stack  in order   
    dPopMemManFileLine( fileName, line ); 
    return (ptr);
}

You will also have to patch the linux and mac platform specific new functions in a similar manner the winMemory.cc mod.

Enjoy
Thats it, use STL to your hearts content (or at least until performance makes you look at pooled allocators and the rest of the 'fun' for using the STL in a game :-) )

Note:
I originally wrote it for some work using TSE, while I haven't tested it recently I see no reason why it shouldn't work out of the box on TSE as well.

About the author

Been a games dev for more years than its comfortable to remember. Have shipped a fair few games, most recently have been Lead Programmer on Brink and Heavenly Sword.

Page«First 1 2 Next»
#21
10/16/2006 (5:53 am)
@Hubertus: Did you get rid of the "Array alloc mismatch" error? In that case, I'd be interested to know how :)

EDIT: Figured it out. Apparently I had "accidently" left out a #define new somewhere... Which resulted in always using the new-allocator, even when the new[]-allocator was supposed to do it's magic. Yay! All of these preprocessor commands around the code really make my head hurt :)

-Trond
#22
03/22/2007 (11:19 am)
I'm using VS2005 and I'm getting these errors when compiling code containing STL:

1>c:\program files\microsoft visual studio 8\vc\include\xlocale(138) : error C2833: 'operator dPushMemManFileLine' is not a recognized operator or type
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(138) : error C2059: syntax error : 'newline'
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(138) : error C2334: unexpected token(s) preceding ':'; skipping apparent function body
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(144) : error C2833: 'operator dPushMemManFileLine' is not a recognized operator or type
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(144) : error C2059: syntax error : 'newline'
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(144) : error C2334: unexpected token(s) preceding ':'; skipping apparent function body
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(151) : error C2039: '_DebugHeapTag_t' : is not a member of 'std'
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(151) : error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(151) : error C2143: syntax error : missing ',' before '&'
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(158) : error C2039: '_DebugHeapDelete' : is not a member of 'std'
1>c:\program files\microsoft visual studio 8\vc\include\xlocale(158) : error C3861: '_DebugHeapDelete': identifier not found

Any ideas?

Thanks.
#23
03/22/2007 (1:34 pm)
Try putting all your STL #includes before any Torque #includes. Then try the other way around. Also try moving your STL #includes from header files to .cc files and then the other way around.

It's crude, I know, but so far I've always found a combination that works.
#24
04/16/2007 (10:22 am)
I am also experiencing the same problems as Igor with VS2005: lots of errors in xlocale. I also tried to incorporate STL code into a fresh Torque installation, but even with the simple example below I get those errors in the xlocale. Reordering the include files also did not fix the problem. Does anyone know another (better) solution than the reordering approach? Does the simple STLTest example work for other people who also use VS2005, Torque 1.4 and the stl_fix? If it works in other environments what else could cause these errors?

#include "core/stl_fix.h"
#include <string>
#include <vector>

class STLTest {

private:
	std::vector<std::string> *simpleStrings;

public:
	void useSTL();	
	
	STLTest() {
		simpleStrings = new std::vector<std::string>();
	}
};
----
#include "stltest/stltest.h"
#include "console/console.h"

void
STLTest::useSTL()
{
	for (std::vector<std::string>::iterator iter = simpleStrings->begin();
		 iter != simpleStrings->end();
		 iter++) 
	{
		Con::printf((*iter).c_str());
	}
}
#25
04/19/2007 (1:16 pm)
@Hubertus
have you been able to get this working? I inserted the stltest code you posted into my TGE 1.3-based project, and it compiled ok.

Exact specifics are

stltest.h:
//#include "core/stl_fix.h"
#include <string>
#include <vector>

class STLTest {

private:
std::vector<std::string> *simpleStrings;

public:
void useSTL();

STLTest() {
simpleStrings = new std::vector<std::string>();
}
};

stltest.cpp:
#include "stltest.h"
#include "console/console.h"

void
STLTest::useSTL()
{
for (std::vector<std::string>::iterator iter = simpleStrings->begin();
iter != simpleStrings->end();
iter++)
{
Con::printf((*iter).c_str());
}
}

and vs2005 was happy:
------ Build started: Project: HFW, Configuration: Release Win32 ------
Compiling...
stltest.cpp
Linking...
HFW - 0 error(s), 0 warning(s)
========== Build: 1 succeeded, 0 failed, 6 up-to-date, 0 skipped ==========

maybe there's a difference betwen Torque 1.3 and 1.4.
#26
04/23/2007 (1:52 am)
Thanks for testing the code, if I try it on a clean 1.4 installation I get

1>------ Build started: Project: Torque Demo, Configuration: Debug Win32 ------
1>Linking...
1>stltest.obj : error LNK2005: "void * __cdecl operator new(unsigned int,void *)" (??2@YAPAXIPAX@Z) already defined in winMemory.obj
1>../example/torqueDemo_DEBUG.exe : fatal error LNK1169: one or more multiply defined symbols found
1>Build log was saved at "file://c:\programs\Torque-test2\SDK\engine\out.VC8.DEBUG\BuildLog.htm"
1>Torque Demo - 2 error(s), 0 warning(s)
========== Build: 0 succeeded, 1 failed, 6 up-to-date, 0 skipped ==========

I think this problem relates to conflicting definitions of the new operator in the Torque memory manager and the visual studio stl implementation. There should be no difference between Torque 1.3 and 1.4 regarding this problem, since Torque 1.3 also uses the memory manager. Geom, did you modify some of your platform specific Torque code or are you using the code that comes with the installation?
#27
04/23/2007 (7:20 am)
Found it! Comment out the definition of operator new in winMemory.cc.

//--------------------------------------
//void* FN_CDECL operator new(dsize_t, void* ptr)
//{
//   return (ptr);
//}

That should fix the linker error.
#28
12/06/2007 (9:58 pm)
Has anyone got this working on Linux using GCC4? Am having some trouble...
#29
12/07/2007 (7:34 am)
doesn't seem to exist with Linux.
#30
06/03/2008 (11:21 pm)
I'm trying this with TGB 1.7.3, getting this error:

2>..\..\source\platformWin32\winMemory.cc(49) : error C3861: 'dPopMemManFileLine': identifier not found

Now winMemory.cc includes platformWin32.h, which includes platform.h, which declares dPopMemManFileLine as an extern, which is defined in platformMemory.cc ...

So whats the problem?

Also i'm still getting "new" redefinition errors (from integrating a library that uses stl) which this would supposedly fix, but maybe that will clear up after i fix the first error.


Ugh, now I'm also getting the:
1>C:\Program Files\Microsoft Visual Studio 8\VC\include\crtdbg.h(1122) : error C2833: 'operator dPushMemManFileLine' is not a recognized operator or type
1>C:\Program Files\Microsoft Visual Studio 8\VC\include\crtdbg.h(1122) : error C2059: syntax error : 'newline'
Page«First 1 2 Next»