STL and the TGE memory manager
by Joel Baxter · in Torque Game Engine · 01/06/2002 (9:54 pm) · 65 replies
I'd like to use STL, at least in some of the tools that link with Torque Lib, but there's problems if the TGE memory manager is enabled (as it is by default) in Torque Lib. The answer may just be "don't use the TGE memory manager", which is OK, but maybe there's another solution... here's what I see:
If the TGE memory manager is enabled, "new" is redefined as "new(__FILE__, __LINE__)" in platform.h. This totally wreaks havoc with the STLport header named "new", and may or may not cause problems in the header used in the standard STL implementation for various platforms.
Now, if I put the STL includes before the platform.h include, it solves that particular problem, but it means that STL operations will not be using the TGE memory manager. If I'm using STLport though, I can put platform.h before the STL includes, but try to edit the STLport include files so that the "new" macro doesn't cause problems. Soooo... maybe this particular problem isn't a killer one, since I like the idea of using STLport.
However, that's not the only problem. :-)
{platform_prefix}Memory.cc defines a default placement operator for new. Unfortunately, the header file named "new" defines an inline default placement operator for new. Let's say that I'm using STL in a tool that links with the Torque Lib. Uh-oh, now there are two default placement operator definitions, one in the Torque Lib from {platform_prefix}Memory.cc, and one in the tool object files. Link fails.
For Windows, I can just #define __PLACEMENT_NEW_INLINE in the STLport config header or in my source before the STL includes, and the problem goes away (or at least the thing compiles). However this won't do diddly-squat for building with gcc on Linux, and probably won't do anything in the Mac build either.
What I'm really wondering, then, is why does the TGE code define that default placement operator?... because that may be the only insurmountable obstacle. In the Windows and Linux source, the TGE-defined default placement operator doesn't do anything different from the standard one (don't have a Mac dev environment for comparison, so I dunno if that is also true for the Mac source). I've tried just commenting it out of winMemory.cc and building the Windows versions of Torque Demo and Torque Lib, and they compile and link without problems. So... whazzit there for?
If the TGE memory manager is enabled, "new" is redefined as "new(__FILE__, __LINE__)" in platform.h. This totally wreaks havoc with the STLport header named "new", and may or may not cause problems in the header used in the standard STL implementation for various platforms.
Now, if I put the STL includes before the platform.h include, it solves that particular problem, but it means that STL operations will not be using the TGE memory manager. If I'm using STLport though, I can put platform.h before the STL includes, but try to edit the STLport include files so that the "new" macro doesn't cause problems. Soooo... maybe this particular problem isn't a killer one, since I like the idea of using STLport.
However, that's not the only problem. :-)
{platform_prefix}Memory.cc defines a default placement operator for new. Unfortunately, the header file named "new" defines an inline default placement operator for new. Let's say that I'm using STL in a tool that links with the Torque Lib. Uh-oh, now there are two default placement operator definitions, one in the Torque Lib from {platform_prefix}Memory.cc, and one in the tool object files. Link fails.
For Windows, I can just #define __PLACEMENT_NEW_INLINE in the STLport config header or in my source before the STL includes, and the problem goes away (or at least the thing compiles). However this won't do diddly-squat for building with gcc on Linux, and probably won't do anything in the Mac build either.
What I'm really wondering, then, is why does the TGE code define that default placement operator?... because that may be the only insurmountable obstacle. In the Windows and Linux source, the TGE-defined default placement operator doesn't do anything different from the standard one (don't have a Mac dev environment for comparison, so I dunno if that is also true for the Mac source). I've tried just commenting it out of winMemory.cc and building the Windows versions of Torque Demo and Torque Lib, and they compile and link without problems. So... whazzit there for?
#2
-Tim aka Spock
01/09/2002 (12:39 am)
So are we just screwed if we need to use the STL? Currently I just wrote my own Link List functions and didn't even try the STL since you said it wouldn't work.-Tim aka Spock
#3
I'd really like to use the STL (preferably without messing around to much with the existing memory code).
01/09/2002 (7:12 am)
I'm also interested in this topic.I'd really like to use the STL (preferably without messing around to much with the existing memory code).
#4
Currently, I do any STL-related includes before including platform.h. That solves the problem of "new" being redefined as a macro. It also means that STL operations will not use the TGE memory manager even if NO_MEMORY_MANAGER was undefined, but c'est la vie. Once I get things working in this state, I may investigate the possible solutions mentioned above to make STL ops use the TGE memory manager, or maybe not.
Also as mentioned above, I've had to comment out the default placement operator declaration in the {platform-prefix}Memory.cc files, like winMemory.cc. This is the "operator new(dsize_t, void* ptr)" declaration, BTW. Like I said, I don't see a reason for that to be there, but it sorta makes me nervous to remove it since I don't know why it was there to begin with. I was hoping one of the staff that ride herd on these forums had either written the memory manager or knew about it.
---
It's interesting to note that they must have had this same problem when writing map2dif. I noticed a couple of days ago that in exportGeometry.cc (which uses) they #defined __PLACEMENT_NEW_INLINE at the top of the file, one of the possible solutions I listed for the new-macro problem. Obviously that isn't a crossplatform solution though.
01/09/2002 (11:18 am)
You can probably use STL. I've started writing it in, and things compile, although I haven't run the code yet.Currently, I do any STL-related includes before including platform.h. That solves the problem of "new" being redefined as a macro. It also means that STL operations will not use the TGE memory manager even if NO_MEMORY_MANAGER was undefined, but c'est la vie. Once I get things working in this state, I may investigate the possible solutions mentioned above to make STL ops use the TGE memory manager, or maybe not.
Also as mentioned above, I've had to comment out the default placement operator declaration in the {platform-prefix}Memory.cc files, like winMemory.cc. This is the "operator new(dsize_t, void* ptr)" declaration, BTW. Like I said, I don't see a reason for that to be there, but it sorta makes me nervous to remove it since I don't know why it was there to begin with. I was hoping one of the staff that ride herd on these forums had either written the memory manager or knew about it.
---
It's interesting to note that they must have had this same problem when writing map2dif. I noticed a couple of days ago that in exportGeometry.cc (which uses
#5
01/09/2002 (10:50 pm)
Well, it all seems to be working as expected. Guess I'll just forge ahead with that default placement operator commented out.
#6
01/10/2002 (6:47 am)
OK, thanks for the info!
#7
BTW... do the GG staff/coders participate in these forums? (I'm new in town.)
01/15/2002 (12:29 pm)
One *bump*, for a last try at getting any other input on why that function was there.BTW... do the GG staff/coders participate in these forums? (I'm new in town.)
#8
-Tim aka Spock
01/17/2002 (11:01 pm)
For anyone needing the Vector class in the STL, i just found that there is an implementation of a vector class in torque but it doesnt not initialize, construct, or deconstruct classes. It is only meant for simple types. This is better than nothing. It is located in engine/core/tVector.cc and tVector.h.-Tim aka Spock
#9
Joel's system above seems to work... Thanks Joel
Will report if it doesn't happen for me..
Why is this like this?
-J
08/01/2002 (11:59 pm)
Hm... I ran into this, killed some productive time... Joel's system above seems to work... Thanks Joel
Will report if it doesn't happen for me..
Why is this like this?
-J
#10
That would work, wouldn't it?
08/05/2002 (1:16 pm)
Why don't you guys just use your STL containers with custom allocator objects that use the Torque Memory manager?That would work, wouldn't it?
#13
02/18/2003 (11:47 am)
I ran into this problem with DirectX 9.0 actually. The Stack Aligned Matricies throw a fit even if the memory manager is disabled. I ended up going into the DX includes and commenting stuff out :-P
#14
03/10/2003 (1:53 am)
Isn't it about time to do something to solve this issue with the Torque Memory Manager? Wouldn't the "TGE 2" engine restructuration process (that will be starting soon hopefully...) a good reason to get rid of this problem? It's pretty limitating not to be able to use STL for *anything*...
#16
If you look in platform.h, you'll find this template:
template
inline T* constructInPlace(T* p)
{
return new(p) T;
}
Searching on constructInPlace in the current CVS head turns up 57 uses. By and
large I can't see much of a reason for using placement new at all in an
engine like this; some of the uses are downright scary and probably not
altogether portable.
I'm curious to know what sort of benchmarking has been done to compare the
custom memory manager with the compiler-supplied one. My own benchmarking
with Windows memory allocation has it being pretty good. It's also interesting
to note that the custom memory manager uses a red-black tree for it's
data structure, which is the typical data structure used for the STL map. Too
much reinventing the wheel going on...
03/12/2003 (10:29 pm)
Regarding the placement new operator defined in winMemory.cc...If you look in platform.h, you'll find this template:
template
inline T* constructInPlace(T* p)
{
return new(p) T;
}
Searching on constructInPlace in the current CVS head turns up 57 uses. By and
large I can't see much of a reason for using placement new at all in an
engine like this; some of the uses are downright scary and probably not
altogether portable.
I'm curious to know what sort of benchmarking has been done to compare the
custom memory manager with the compiler-supplied one. My own benchmarking
with Windows memory allocation has it being pretty good. It's also interesting
to note that the custom memory manager uses a red-black tree for it's
data structure, which is the typical data structure used for the STL map. Too
much reinventing the wheel going on...
#17
03/15/2003 (2:37 am)
*bump* 
#18
"Bumping" this thread will not get you the action you want.
Jeff Tunnell GG
03/17/2003 (9:50 am)
Steffan,"Bumping" this thread will not get you the action you want.
Jeff Tunnell GG
#19
03/17/2003 (10:03 am)
Well it's not like he's slacking. He's also kvetching in #garagegames on irc.maxgaming.net.
#20
I don't *know* how to fix it and what the consequences would be if I disable the Torque Memory Manager... I couldn't even get the engine to compile yet if I disable it... :/
So I guess I have to write my own container classes and forget about STL and Torque...
thx
03/17/2003 (10:10 am)
well, I only want some GG comment on this... people are complaining about this problem for over a year now in various threads, and no GG employee did ever respond yet AFAIK...I don't *know* how to fix it and what the consequences would be if I disable the Torque Memory Manager... I couldn't even get the engine to compile yet if I disable it... :/
So I guess I have to write my own container classes and forget about STL and Torque...
thx
Torque 3D Owner Joel Baxter