STL
by Dean Calver · 04/03/2005 (9:47 am) · 30 comments
2009 Update!!! TGEA and I expect Torque3D work out the box with STL now, so this nastiness should be a thing of the past thankfully :)
<a href="/static/pg/resource/0.stl_fix.h">Download Code File</a>
STL fix
The STL is a massive collection of classes that all use a pattern called allocators for all memory allocation stuff. The particular type of macros used by the Torque engine memory managment system cause most (certainly the windows and linux standard versions) STL allocators to not compile.
One 'solution' is to disable the torque memory manager completely but the torque memory manager does lots of good things, especially in the realms of debugging. It will save your arse at least once before you ship your game, so disabling it is BAD.
So a better fix is while keeping the existing torque memory manager replace the provided STL allocators with ones that work 'nicely' with torque.
This proves to be a path of darkness and true system level hackery as in there infinite wisdom, the C++ standards commitee have proved no way of replacing the default allocator (you can provide optionally different allocators, which is useful for pooled allocation but not replace the default allocator).
The problem is that each platform and compiler use different implementation of std::allocator so I needed to figure out how to get rid of each one in turn.
This major addition is stl_fix.h, its needs including before any stl header is included and then is just works... As new platform, compilers etc. come in stl_fix.h will need updating but I think I've covered the current windows side of things and probably Linux. Mac I don't know as I don't have access to one.
Well it almost "just works", you see the macro form of new that torque uses still upsets some of the more obscure parts of some implementations of STL. The only solution is to patch torque, the actual patch is no more than a few lines and loses no functionality.
I tried for a long time to come up with a solution that didn't require a torque patch but in the end I had to fallback to a patch.
How to get it worked
Thomus Lund's excellent GameSWG resource needed to use stl_fix, so helped sort out a few bugs and he also wrote up the instructions for how to use it. As Thomus's explaniation is so good, I've basically copied his instructions into this seperate resource.
Attached to this resource is stl_fix.h this should be copied to your engine/core directory. Include this before any stl headers.
Open up platform/platform.h
Find this in Line 396
Open the file platform/platformMemory.cc
Find this in line 1080
open platformWin32/winMemory.cc
Find this at Line 41
You will also have to patch the linux and mac platform specific new functions in a similar manner the winMemory.cc mod.
Enjoy
Thats it, use STL to your hearts content (or at least until performance makes you look at pooled allocators and the rest of the 'fun' for using the STL in a game :-) )
Note:
I originally wrote it for some work using TSE, while I haven't tested it recently I see no reason why it shouldn't work out of the box on TSE as well.
<a href="/static/pg/resource/0.stl_fix.h">Download Code File</a>
STL fix
The STL is a massive collection of classes that all use a pattern called allocators for all memory allocation stuff. The particular type of macros used by the Torque engine memory managment system cause most (certainly the windows and linux standard versions) STL allocators to not compile.
One 'solution' is to disable the torque memory manager completely but the torque memory manager does lots of good things, especially in the realms of debugging. It will save your arse at least once before you ship your game, so disabling it is BAD.
So a better fix is while keeping the existing torque memory manager replace the provided STL allocators with ones that work 'nicely' with torque.
This proves to be a path of darkness and true system level hackery as in there infinite wisdom, the C++ standards commitee have proved no way of replacing the default allocator (you can provide optionally different allocators, which is useful for pooled allocation but not replace the default allocator).
The problem is that each platform and compiler use different implementation of std::allocator so I needed to figure out how to get rid of each one in turn.
This major addition is stl_fix.h, its needs including before any stl header is included and then is just works... As new platform, compilers etc. come in stl_fix.h will need updating but I think I've covered the current windows side of things and probably Linux. Mac I don't know as I don't have access to one.
Well it almost "just works", you see the macro form of new that torque uses still upsets some of the more obscure parts of some implementations of STL. The only solution is to patch torque, the actual patch is no more than a few lines and loses no functionality.
I tried for a long time to come up with a solution that didn't require a torque patch but in the end I had to fallback to a patch.
How to get it worked
Thomus Lund's excellent GameSWG resource needed to use stl_fix, so helped sort out a few bugs and he also wrote up the instructions for how to use it. As Thomus's explaniation is so good, I've basically copied his instructions into this seperate resource.
Attached to this resource is stl_fix.h this should be copied to your engine/core directory. Include this before any stl headers.
Open up platform/platform.h
Find this in Line 396
extern void* FN_CDECL operator new(dsize_t size, const char*, const U32); extern void* FN_CDECL operator new[](dsize_t size, const char*, const U32); extern void FN_CDECL operator delete(void* ptr); extern void FN_CDECL operator delete[](void* ptr); #define new new(__FILE__, __LINE__)and change to this
extern void* FN_CDECL operator new(dsize_t size); extern void* FN_CDECL operator new[](dsize_t size); extern void FN_CDECL operator delete(void* ptr); extern void FN_CDECL operator delete[](void* ptr); extern void dPopMemManFileLine( const char*&, U32& ); extern int dPushMemManFileLine( const char*, const U32 ); #define new dPushMemManFileLine( __FILE__, __LINE__ ) ? 0 : new
Open the file platform/platformMemory.cc
Find this in line 1080
#if !defined(TORQUE_DISABLE_MEMORY_MANAGER)
// Manage our own memory, add overloaded memory operators and functions
void* FN_CDECL operator new(dsize_t size, const char* fileName, const U32 line)
{
return Memory::alloc(size, false, fileName, line);
}
void* FN_CDECL operator new[](dsize_t size, const char* fileName, const U32 line)
{
return Memory::alloc(size, true, fileName, line);
}
void* FN_CDECL operator new(dsize_t size)
{
return Memory::alloc(size, false, NULL, 0);
}
void* FN_CDECL operator new[](dsize_t size)
{
return Memory::alloc(size, true, NULL, 0);
}and change to this#if !defined(TORQUE_DISABLE_MEMORY_MANAGER)
static const int NUM_NEW_STACK_SIZE = 256;
static struct { const char* filename; U32 line;}
g_NewStackMemDebug[NUM_NEW_STACK_SIZE];
static int g_CurStack = -1;
int dPushMemManFileLine( const char* filename, U32 line )
{
if(g_CurStack >= NUM_NEW_STACK_SIZE )
return 0;
g_CurStack++;
g_NewStackMemDebug[g_CurStack].filename = filename;
g_NewStackMemDebug[g_CurStack].line = line;
return 0; // needed for the new passthrough trick
}
void dPopMemManFileLine( const char*& filename, U32& line )
{
if( g_CurStack < 0 )
{
filename = 0;
line = 0;
return;
} else
{
filename = g_NewStackMemDebug[g_CurStack].filename;
line = g_NewStackMemDebug[g_CurStack].line;
g_CurStack--;
}
}
// Manage our own memory, add overloaded memory operators and functions
void* FN_CDECL operator new(dsize_t size)
{
const char* fileName = 0;
U32 line;
dPopMemManFileLine( fileName, line );
return Memory::alloc(size, false, fileName, line);
}
void* FN_CDECL operator new[](dsize_t size)
{
const char* fileName = 0;
U32 line;
dPopMemManFileLine( fileName, line );
return Memory::alloc(size, true, fileName, line);
}open platformWin32/winMemory.cc
Find this at Line 41
void* FN_CDECL operator new(dsize_t, void* ptr)
{
return (ptr);
}and change to thisvoid* FN_CDECL operator new(dsize_t, void* ptr)
{
const char* fileName = 0;
U32 line;
// dummy does nothing except keep the stack in order
dPopMemManFileLine( fileName, line );
return (ptr);
}You will also have to patch the linux and mac platform specific new functions in a similar manner the winMemory.cc mod.
Enjoy
Thats it, use STL to your hearts content (or at least until performance makes you look at pooled allocators and the rest of the 'fun' for using the STL in a game :-) )
Note:
I originally wrote it for some work using TSE, while I haven't tested it recently I see no reason why it shouldn't work out of the box on TSE as well.
About the author
Been a games dev for more years than its comfortable to remember. Have shipped a fair few games, most recently have been Lead Programmer on Brink and Heavenly Sword.
#2
04/06/2005 (12:10 pm)
This is the fix Pat mentioned (I'll send him an email). In the end I needed to patch the memory manager syntax to work with some of more obscure parts.
#3
The link seems to be broken though.. Any mirrors?
04/10/2005 (9:38 am)
This is great, I really needed that..The link seems to be broken though.. Any mirrors?
#4
04/12/2005 (12:34 pm)
stl_fix.h is on my server at http://rattie.demon.co.uk/stl_fix.h
#5
LList
Vector
SparseArray
MatrixF
BitMatrix, BitSet32, BitVector, BitVectorW
MatrixF
Lots of Point and Rectangle classes
and probably a whole lot of stuff I missed.
04/12/2005 (9:32 pm)
Before you feel like you -have- to have the STL, keep in mind that Torque has a lot of useful data structures already implemented.LList
Vector
SparseArray
MatrixF
BitMatrix, BitSet32, BitVector, BitVectorW
MatrixF
Lots of Point and Rectangle classes
and probably a whole lot of stuff I missed.
#6
04/22/2005 (10:11 am)
Thanks, Great resource
#7
C:\isi\1763-champlain\synapsegaming_lightingpack-1.3.0\tools\max2dtsexporter\shapemimic.cc(3325) : error C2664: 'Vector::push_back' : cannot convert parameter 1 from 'TSDecalMesh *' to 'TSMesh *const & '
with
[
T=TSMesh *
]
Reason: cannot convert from 'TSDecalMesh *' to 'TSMesh *const '
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
It was working perfectly before applying the stl fixes
04/27/2005 (1:03 pm)
I'm getting this compile error after apply the STL fixes as described above, anyone else know why?C:\isi\1763-champlain\synapsegaming_lightingpack-1.3.0\tools\max2dtsexporter\shapemimic.cc(3325) : error C2664: 'Vector
with
[
T=TSMesh *
]
Reason: cannot convert from 'TSDecalMesh *' to 'TSMesh *const '
Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
It was working perfectly before applying the stl fixes
#8
www.garagegames.com/docs/tge/engine/hierarchy.php
If they ARE related, perhaps figuring out why Doxygen is confused will provide you with an answer as to why STL is confused.
Strange. TSMesh's mesh-type enum lists decals, so you'd think they're related. Very Strange.
Is it privately inherited? Something weird like that?
If all else fails, an old-style () cast should do the trick. C-style casts can force any pointer type to any other pointer type (even when thats a Very Bad idea):
push_back( (const TSMesh*) myDecalPtr );
05/18/2005 (10:53 am)
According to Doxygen, TSMesh and TSDecalMesh aren't related. www.garagegames.com/docs/tge/engine/hierarchy.php
If they ARE related, perhaps figuring out why Doxygen is confused will provide you with an answer as to why STL is confused.
Strange. TSMesh's mesh-type enum lists decals, so you'd think they're related. Very Strange.
Is it privately inherited? Something weird like that?
If all else fails, an old-style () cast should do the trick. C-style casts can force any pointer type to any other pointer type (even when thats a Very Bad idea):
push_back( (const TSMesh*) myDecalPtr );
#9
Anyone get the same problem ?
07/18/2005 (7:20 am)
I can't get the stl_fix.h , when I click the link it return to the ressource page. Anyone get the same problem ?
#10
first
d:\Torque\SDK\engine\gameswf\gameswf_action.cpp(1371): error C2664:
'gameswf::as_object::set_member' : cannot convert parameter 2 from 'void (const gameswf::fn_call &)' to 'const gameswf::as_value &'
12/06/2005 (3:17 am)
i am getting this errorfirst
d:\Torque\SDK\engine\gameswf\gameswf_action.cpp(1371): error C2664:
'gameswf::as_object::set_member' : cannot convert parameter 2 from 'void (const gameswf::fn_call &)' to 'const gameswf::as_value &'
#11
02/13/2006 (5:49 am)
some error here with TGE 1.4 and VC6.0
#12
05/09/2006 (11:53 pm)
Its work with VS Express 2005! :)
#14
void* FN_CDECL operator new(dsize_t, void* ptr)
{
return (ptr);
}
and replaced it with an #include.
I realize this is a bit of a hack, and better c++ coders than I will probably cringe, but it got the job done for me, which everything else so far had only pushed me in the right direction.
06/07/2006 (1:04 pm)
Searching GG almost endlessly to bypass this problem, I've found several solutions. First and foremost is the above stl_fix.h. However, when following the code directions above, I had trouble. So, what I ended up doing is adding the stl_fix.h before my STL code, and in winMemory.cc I removed the new operatorvoid* FN_CDECL operator new(dsize_t, void* ptr)
{
return (ptr);
}
and replaced it with an #include
I realize this is a bit of a hack, and better c++ coders than I will probably cringe, but it got the job done for me, which everything else so far had only pushed me in the right direction.
#15
"Array Alloc mismatch"
in platform/platformmemory.cc on line 1038 which says
AssertFatal(((bool)((hdr->flags & Array)==Array))==array, avar("Array alloc mismatch. "));
Has anyone else already dealt with this issue? I found one posting but no answer in one other resource and no postings in the forum at all.
06/12/2006 (9:54 am)
For me (Torque 1.4, MSVC 7.0) compilation was no problem, but the debug build won't run due to a violated assertion. I get an"Array Alloc mismatch"
in platform/platformmemory.cc on line 1038 which says
AssertFatal(((bool)((hdr->flags & Array)==Array))==array, avar("Array alloc mismatch. "));
Has anyone else already dealt with this issue? I found one posting but no answer in one other resource and no postings in the forum at all.
#16
07/24/2006 (11:00 am)
@Michigan State -> You were running into the issue where it errors at linking, saying that the operator new is already defined? Does your solution still make use of Torque's memory management? Have you had any issue while using this technique? Thanks! :D
#17
can you please send it to me sa6ry@sa6ry.com ?
or better upload it to any server.
thanks in advance
09/04/2006 (2:44 am)
stl_fix.h link seems to be broken ( both the garagegame link and the one posted by Dean Calvercan you please send it to me sa6ry@sa6ry.com ?
or better upload it to any server.
thanks in advance
#18
Like Mostafa Ashour said, could someone please publish it or send it to my email: e0105181@student.tuwien.ac.at ?
best regards
09/12/2006 (9:27 am)
I would also need this resource.Like Mostafa Ashour said, could someone please publish it or send it to my email: e0105181@student.tuwien.ac.at ?
best regards
#20
Thank goodness, too, I really can't live without STL.
The problem's not totally gone though. You still have to play around with #include file ordering bit. But so far, I've always managed to find an ordering that works.
09/19/2006 (8:20 pm)
Visual Studio 2005 has greatly improved this situation. With it I've been using using STL + Torque for several months now, without the stl_fix.Thank goodness, too, I really can't live without STL.
The problem's not totally gone though. You still have to play around with #include file ordering bit. But so far, I've always managed to find an ordering that works.

Associate Tom Spilman
Sickhead Games