Initial Atlas Terrain Checkin (Pretty Pictures In Here!)
by Ben Garney · in Torque Game Engine Advanced · 05/27/2005 (6:36 pm) · 249 replies
Hey guys,
I just committed the first public version of the new TSE terrain, codenamed Atlas. This is a fully modern, scalable terrain engine. Check the docs (updated with a whole new chapter :) for details.
This is a private release to our beloved TSE licensees. Congrats, guys, you get first shot at this new toy. :) We'll be rolling out a full launch of Milestone 2 - this was the last piece - next week when we all get back from Memorial Day of Weekend. I'd appreciate it ever so much if you'd keep on the quiet side about this until then!
The toolchain is currently a bit on the basic side. A fuller toolchain will be rolled out as part of MS3.
Please take a moment to give it a try. Feedback is welcome - post it here, in a new thread on this forum, or mail it to me. There's a lot more work to be done, but this represents a basic usable point for people to work from. :)
You can get this release by doing a CVS update, of course.
And finally... A rundown on the features:
Lightweight CPU Usage
Atlas uses less than 10% of CPU time per frame to update and render. Compare this with approximately 30% for legacy terrain.
Massive Landscapes
Supports landscapes of arbitrary size, bounded only by disk space. The current system supports approximately 10 quinquagintillion (or ten thousand quintillion vigintillion for those Brits in the audience, or 10154 chunks on the base level).
Vast View Distances
Can quickly render unlimited view distances, bounded only by the working range of the video card's Z-Buffer. Atlas can achieve 100+ fps on a midrange Radeon, with a view distance of approximately 4,000 meters. The legacy terrain is not even capable of rendering this view distance by default!
Unique Textures
Atlas supports a single massive texture for your entire terrain! Every texel is unique. Support for extremely large textures (in excess of 32,768 by 32,768 pixels) on all hardware. Of course, all this is streamed, so you only pay memory cost for the texture detail near you. A future release will support on-the-fly texture blending like the legacy terrain.
Streaming Data
Data is streamed from the disc as needed, so you can have very large terrains without exceeding your memory. Both geometry and texture data are loaded on the fly.
Accurate Collision
What is rendered is what is collided against, so you get full interaction support for anything Atlas can render - overhangs, gaps, ditches, and so on. As the toolset for this engine matures, it will be possible to actually create such terrains. But the runtime as of now supports this sort of topology.
Compressed Textures
Atlas fully supports industry-standard DXT texture compression. It compresses offline for maximum quality, pregenerates mipmaps for minimal CPU usage at runtime, then streams it directly into GPU memory.
Ok, I'm going to a party. I'll be back online later tonight to check up on things, and will be around during the weekend as well. Thanks for being so patient with me! :)
#42
I've got problems too. Tried creating a .chu file from a raw file exported in Terragen, but I get loads of warnings and a suggestion error about decreasing the QuadTree. I'm not sure you can use a jpg to create the terrain data yet, as far as I understood it only supported .raw files.
My .chu file is created but cannot be loaded, and using a .jpg (like you) it crashes before it even is created. Hope that helps.
05/28/2005 (11:03 am)
Quote:
Yep, no large terrain for me either... did it get taken down?
I've tried generating my own terrain from an image created in photoshop, has anyone else created thier own .chu files yet and if so, can you paste the console output you got?
I'm getting past the header creation, but then its choking.
I've got problems too. Tried creating a .chu file from a raw file exported in Terragen, but I get loads of warnings and a suggestion error about decreasing the QuadTree. I'm not sure you can use a jpg to create the terrain data yet, as far as I understood it only supported .raw files.
My .chu file is created but cannot be loaded, and using a .jpg (like you) it crashes before it even is created. Hope that helps.
#43
I wonder if I did something wrong.
I'll try a tiny dataset first and see if that helps.
05/28/2005 (11:50 am)
Naah, I'm using a .RAW file I created with photoshop (photoshop can write raw 16 bit greyscale images too), it in theory *should* be ok. But its not getting futher than the TOC.I wonder if I did something wrong.
I'll try a tiny dataset first and see if that helps.
#44
I'm only gonna answer questions that directly relate to the terrain here... Otherwise this post will be twice as long. If you've got a question I didn't answer, try posting in a new thread!
It looks like there is indeed some memory problem with it. I'll track that down later today. It's definitely an intermittent issue as I've run the terrain a great number of times on my system with no ill effects (including with validation of the memory on). You people experiencing memory manager related crashes, are you using VC6 or VC7?
@Ben Curley: Actually, those structures SHOULD be always the same size. 64 bytes. Are you running on 64bit by any chance? I've updated those asserts to give a more meaningful error, too... Disabled them as well since they seem to give false errors. Can anyone who had this problem tell me what sizeof(Memory::Header) and sizeof(Memory::FreeHeader) and sizeof(Memory::AllocatedHeader) are coming out to?
Strange the webpage is conking out for some. It's an absolutely dead simple plain HTML page. It might be the server is a bit unhappy that so much data is getting downloaded from it, but it's hard to say. Or maybe it's an IE specific problem. I've mailed Benjamin to fix it, so it'll probably get dealt with sometime today.
@Josh: Whoops. I was testing with a different player class and forgot to check with the new one when I did the final merge against CVS.
@Tom: The runtime and file formats support it... The initial from-heightfield mesh generator does not, unfortunately.
@Phil: What size is the .RAW file? What are the parameters for your chunk file generation?
So, changes I've made since my last message (do update to get these changes):
Re-enabled player collision.
Disabled the memory asserts, as well as giving them more explanative error text.
Look forward to further feedback. You've already found a few issues I'm glad got out now, instead of in a week. :)
05/28/2005 (12:21 pm)
Hey guys... Glad to see Atlas is getting played with. :)I'm only gonna answer questions that directly relate to the terrain here... Otherwise this post will be twice as long. If you've got a question I didn't answer, try posting in a new thread!
It looks like there is indeed some memory problem with it. I'll track that down later today. It's definitely an intermittent issue as I've run the terrain a great number of times on my system with no ill effects (including with validation of the memory on). You people experiencing memory manager related crashes, are you using VC6 or VC7?
@Ben Curley: Actually, those structures SHOULD be always the same size. 64 bytes. Are you running on 64bit by any chance? I've updated those asserts to give a more meaningful error, too... Disabled them as well since they seem to give false errors. Can anyone who had this problem tell me what sizeof(Memory::Header) and sizeof(Memory::FreeHeader) and sizeof(Memory::AllocatedHeader) are coming out to?
Strange the webpage is conking out for some. It's an absolutely dead simple plain HTML page. It might be the server is a bit unhappy that so much data is getting downloaded from it, but it's hard to say. Or maybe it's an IE specific problem. I've mailed Benjamin to fix it, so it'll probably get dealt with sometime today.
@Josh: Whoops. I was testing with a different player class and forgot to check with the new one when I did the final merge against CVS.
@Tom: The runtime and file formats support it... The initial from-heightfield mesh generator does not, unfortunately.
@Phil: What size is the .RAW file? What are the parameters for your chunk file generation?
So, changes I've made since my last message (do update to get these changes):
Re-enabled player collision.
Disabled the memory asserts, as well as giving them more explanative error text.
Look forward to further feedback. You've already found a few issues I'm glad got out now, instead of in a week. :)
#45
05/28/2005 (12:34 pm)
Is it just me that thinks the texture file should be called a .pika file?
#46
This is what I typed. What happends is that the engine crashes and kills the bloody stack so I can't debug it :/
It's a normal JPG with compression set to 12.
Anyone had this issue yet?
Btw. Ben Garney, I'm using VC6 to compile and the October SDK from last year. I have the crashes/assert at startup issues 1 time out of 4 aprox.
05/28/2005 (12:35 pm)
GenerateTQT(); is broken somewhere. I used a 512x512 normal JPG and tried to convert it.Quote:
generateTQT("terrain_water_demo/data/terrains/desert.jpg", "terrain_water_demo/data/terrains/desert.tqt", 5, 512);
This is what I typed. What happends is that the engine crashes and kills the bloody stack so I can't debug it :/
It's a normal JPG with compression set to 12.
Anyone had this issue yet?
Btw. Ben Garney, I'm using VC6 to compile and the October SDK from last year. I have the crashes/assert at startup issues 1 time out of 4 aprox.
#47
05/28/2005 (12:37 pm)
Tried a smaller .raw file from photoshop worked fine. The original was 2049x2049 and I used a 513x513 and it worked ok.
#48
The structures, at least in the copy I am pulling down from CVS, are defo not the same size. I have pulled the structures out of the platformmemory.cpp file and put them in here. As you will see the FreeHeader structure contains an extra 16 bytes.
struct Header
{
// doubly linked list of allocated and free blocks -
// contiguous in memory.
#ifdef DEBUG_GUARD
U32 preguard[4];
#endif
Header *next;
Header *prev;
dsize_t size;
U32 flags;
#ifdef DEBUG_GUARD
U32 unused[4];
U32 postguard[4];
#endif
};
struct AllocatedHeader
{
#ifdef DEBUG_GUARD
U32 preguard[4];
#endif
Header *next;
Header *prev;
dsize_t size;
U32 flags;
#ifdef DEBUG_GUARD
// an allocated header will only have this stuff if debug_guard
U32 line;
U32 allocNum;
const char *fileName;
U32 realSize;
U32 postguard[4];
#endif
};
struct FreeHeader
{
#ifdef DEBUG_GUARD
U32 preguard[4];
#endif
Header *next;
Header *prev;
dsize_t size;
U32 flags;
// since a free header has at least one cache line (16 bytes)
// we can tag some more stuff on:
FreeHeader *nextQueue; // of the same size
FreeHeader *prevQueue; // doubly linked
TreeNode *treeNode; // which tree node we're coming off of.
U32 guard;
#ifdef DEBUG_GUARD
U32 postguard[4];
#endif
};
If I fix this little problem then I never seem to have problems with the asserts.
05/28/2005 (12:38 pm)
Ben,The structures, at least in the copy I am pulling down from CVS, are defo not the same size. I have pulled the structures out of the platformmemory.cpp file and put them in here. As you will see the FreeHeader structure contains an extra 16 bytes.
struct Header
{
// doubly linked list of allocated and free blocks -
// contiguous in memory.
#ifdef DEBUG_GUARD
U32 preguard[4];
#endif
Header *next;
Header *prev;
dsize_t size;
U32 flags;
#ifdef DEBUG_GUARD
U32 unused[4];
U32 postguard[4];
#endif
};
struct AllocatedHeader
{
#ifdef DEBUG_GUARD
U32 preguard[4];
#endif
Header *next;
Header *prev;
dsize_t size;
U32 flags;
#ifdef DEBUG_GUARD
// an allocated header will only have this stuff if debug_guard
U32 line;
U32 allocNum;
const char *fileName;
U32 realSize;
U32 postguard[4];
#endif
};
struct FreeHeader
{
#ifdef DEBUG_GUARD
U32 preguard[4];
#endif
Header *next;
Header *prev;
dsize_t size;
U32 flags;
// since a free header has at least one cache line (16 bytes)
// we can tag some more stuff on:
FreeHeader *nextQueue; // of the same size
FreeHeader *prevQueue; // doubly linked
TreeNode *treeNode; // which tree node we're coming off of.
U32 guard;
#ifdef DEBUG_GUARD
U32 postguard[4];
#endif
};
If I fix this little problem then I never seem to have problems with the asserts.
#49
05/28/2005 (12:38 pm)
Same as Phil btw. My 4097x4097 terrain made the engine crash, while the 513x513 one didnt.
#50
Nice terrain! :)
A few improvements to suggest...
The scintillation when using a high noise detail texture on distant hills etc looks a bit strange. Maybe blend out the detail texture based on distance?
I tried using the large texture with a water block and the water block gets culled rather strangely. Make the water 4096x4096 and then look around at different angles it gets culled out. Better culling on the water would be useful so that if ANY of its visible it stays so.
But that bugger is BIG :)
Works fine for me... on a ti4600 card.
Fast too!
05/28/2005 (12:53 pm)
Hey Ben!Nice terrain! :)
A few improvements to suggest...
The scintillation when using a high noise detail texture on distant hills etc looks a bit strange. Maybe blend out the detail texture based on distance?
I tried using the large texture with a water block and the water block gets culled rather strangely. Make the water 4096x4096 and then look around at different angles it gets culled out. Better culling on the water would be useful so that if ANY of its visible it stays so.
But that bugger is BIG :)
Works fine for me... on a ti4600 card.
Fast too!
#51
error("GENERATING TERRAIN");
generateChunkFileFromRaw16(
"terrain_water_demo/data/terrains/test.raw",
512, 2.0, 1.0 / 256.0,
"terrain_water_demo/data/terrains/test.chu", 2.0, 6);
error("DONE GENERATING TERRAIN");
that works fine.. with a 2049 .raw I'd use this..
error("GENERATING TERRAIN");
generateChunkFileFromRaw16(
"terrain_water_demo/data/terrains/test.raw",
2048, 2.0, 1.0 / 256.0,
"terrain_water_demo/data/terrains/test.chu", 2.0, 6);
error("DONE GENERATING TERRAIN");
I'll try just slowly increasing the terrain size..
05/28/2005 (12:57 pm)
Here's my test terrain stuff.. works 513 but not 4097error("GENERATING TERRAIN");
generateChunkFileFromRaw16(
"terrain_water_demo/data/terrains/test.raw",
512, 2.0, 1.0 / 256.0,
"terrain_water_demo/data/terrains/test.chu", 2.0, 6);
error("DONE GENERATING TERRAIN");
that works fine.. with a 2049 .raw I'd use this..
error("GENERATING TERRAIN");
generateChunkFileFromRaw16(
"terrain_water_demo/data/terrains/test.raw",
2048, 2.0, 1.0 / 256.0,
"terrain_water_demo/data/terrains/test.chu", 2.0, 6);
error("DONE GENERATING TERRAIN");
I'll try just slowly increasing the terrain size..
#52
05/28/2005 (1:21 pm)
Hmm, generated 512,1024 and 2048 so far... mustve been dodgy .raw saving..
#53
I think you might be missing something IMPORTANT.
For generateTQT you MUST make sure the leaf sizes match the size of the texture. A tree depth of 5 and a leaf size of 512 means that you're telling it to read in 8192 pixels, which DEFINITELY won't work with a small image. For a 512 image you'd want something like depth=2 and leafSize=256, or maybe just depth=1 and leafSize=512. More or less won't work. I'll add some more informative error messages soonish.
Similarly, for the chunk file generation, pay attention to your polygon budgets. I'm currently a bit unhappy with the decimation algorithm I'm using, as it seems to have a hard time staying on-budget for coarser levels of detail. Depending on the terrain you may have to use a relatively high error threshold, or a deeper tree (each level of detail has level*error for its error threshold, so deeper trees have a higher error threshold at the top than smaller trees - useful for dealing with big terrains).
Generating under debug can be dog slow but might give you some more informative error info. I did about 90% of my tests with a 4097x4097 terrain, so the generation code can definitely handle that.
The number one reason for failed generations in my experience has been a bad error threshold - the decimator exceeds 65536 verts and asserts.
@Ben Curley: Aw nuts. You're right. :) How foolish of me... and how mysterious I never saw the crash. Thanks for looking into this, definitely clears things up.
@Phil: Yeah, definitely want to do some work on the texturing. Probably as part of my blender implementation, I'll also do a bit of work on the detail texturing and related issues. I have some ideas that ought to make it look good... We'll see how they go. :)
05/28/2005 (1:22 pm)
Hey content-generation guys,I think you might be missing something IMPORTANT.
For generateTQT you MUST make sure the leaf sizes match the size of the texture. A tree depth of 5 and a leaf size of 512 means that you're telling it to read in 8192 pixels, which DEFINITELY won't work with a small image. For a 512 image you'd want something like depth=2 and leafSize=256, or maybe just depth=1 and leafSize=512. More or less won't work. I'll add some more informative error messages soonish.
Similarly, for the chunk file generation, pay attention to your polygon budgets. I'm currently a bit unhappy with the decimation algorithm I'm using, as it seems to have a hard time staying on-budget for coarser levels of detail. Depending on the terrain you may have to use a relatively high error threshold, or a deeper tree (each level of detail has level*error for its error threshold, so deeper trees have a higher error threshold at the top than smaller trees - useful for dealing with big terrains).
Generating under debug can be dog slow but might give you some more informative error info. I did about 90% of my tests with a 4097x4097 terrain, so the generation code can definitely handle that.
The number one reason for failed generations in my experience has been a bad error threshold - the decimator exceeds 65536 verts and asserts.
@Ben Curley: Aw nuts. You're right. :) How foolish of me... and how mysterious I never saw the crash. Thanks for looking into this, definitely clears things up.
@Phil: Yeah, definitely want to do some work on the texturing. Probably as part of my blender implementation, I'll also do a bit of work on the detail texturing and related issues. I have some ideas that ought to make it look good... We'll see how they go. :)
#54
I guess the terrain scale has some power there :)
05/28/2005 (1:32 pm)
Oooh, now thats interesting, 32meg .raw file processed down to 84k :)I guess the terrain scale has some power there :)
#55
WOW!!!!!!!! I love new terrain :)
Large test terrain without water and 10000 distance visibility in my FX5700
www.luis-anton.com/images/terrain01.JPG
www.luis-anton.com/images/terrain02.JPG
05/28/2005 (1:35 pm)
WOW!!!!!!!! I love new terrain :)
Large test terrain without water and 10000 distance visibility in my FX5700
www.luis-anton.com/images/terrain01.JPG
www.luis-anton.com/images/terrain02.JPG
#56
Same terrain, spruced up a bit with water and one of Tim Aste's awesome new skyboxes. :)
05/28/2005 (2:01 pm)
Same terrain, spruced up a bit with water and one of Tim Aste's awesome new skyboxes. :)
#57
Has this gone into the CVS?
05/28/2005 (2:16 pm)
Thanks for informing about the leafsize, I thought it was just the size of the texture, not considering the tree depth.Has this gone into the CVS?
#58
05/28/2005 (2:18 pm)
Has what?
#59
If you want to see the water/terrain together, the terrain_water_demo has that.
05/28/2005 (2:18 pm)
Oh, the new skybox? No. Tim needs to eat. :PIf you want to see the water/terrain together, the terrain_water_demo has that.
#60
Did some great work on the terrain Ben, really like it!
Nick
05/28/2005 (2:21 pm)
Really nice! Also, just compiled and ran it with no problems. The camera is jumpy though when moving around but that's already mentioned in the "Call for Bugs" thread. However, just noticed that when standing still above the water, the water itself is jumpy too. Just wanted to let you know.Did some great work on the terrain Ben, really like it!
Nick
Associate Tom Spilman
Sickhead Games