Streaming/Paging question

#1 The way I'm currently looking at interiors is to create a series of smaller, interlocking interiors (much like Tim did with the Castle Pack, but on a much smaller scale). Each of these interiors is an "area node". Each area node contains information for all models, decals, triggers, etc in a tight little package. The area that you are in becomes the "primary node". The engine loads everything in that node. It also loads the secondary nodes (ie. if you have two doors that lead to two more nodes, it loads those nodes and their associated objects. When the player moves from one node to another, a currently loaded list of nodes is checked against a list of needed nodes to load and parses out the ones that aren't needed. It then loads the needed nodes and flushes the nodes that are no longer needed. To make relighting a scene work faster, keeping the interiors small is a necessity, which is also an issue for quick loading. Once I have this working, I'll know more about the dump files that I need to use to save all of the node information both temporarily (while the game is being played) and permanently (when the player clicks save). A huge amount of tweaking will also be necessary.

I've just barely started on this, so I'm scouring the source to see just where I need to make adjustments or rewrites. I've also been looking at Melv's generic render object resource to see if adding a node object would be a good idea. After all, the interior/model/texture loading code is already there. They just need tweaking for this to work the way I want it to.

*edit
There are a number of problems for this implementation. For example, if one were making a GTA clone where you can enter every building, the obvious problem deals with loading the exteriors of buildings (with LOD) in the "outside node", as well as the interiors of the buildings (though perhaps only the foyer) at various levels of detail. That would be huge. Having a large outside map, a load screen, and a large interior would be much more friendly. If you're paging terrain, you'll have 9 areas loaded to move around between and all of the building models and interior nodes.

Since the project I'm working on has a series of interconnected interiors, it's a nice implementation (in theory...we'll see).

#2 The Terrain Manager project basically works around the following premise of a grid of 9 terrain "pages" with your current page at the center:

1 2 3
4 X 5
6 7 8

On entering the game, you not only load the terrain page that the player is "in" (X), but all 8 around it. When you, for example have a "world of the following full size:

A B C D
E F G H
I J K L
M N O P

The client will load in (assuming 'F' is where you are currently).
A B C
E F G
I J K

Now, if you move to the east until you cross over into the 'G' terrain page, the terrain pager will "drop" A, E, and I, and "load" D, H, and L, keeping your current square and all those 1 away "active" at all times, making your currently paged in terrains:

B C D
F G H
J K L

There are a bunch of other things involved, but that is the overall idea.

#3 David, that sounds very interesting. A demo would be very cool to see.

#4 @Josh
I'm excited to see it working as well. I don't think that working with loading and unloading interiors will be that difficult. It's relighting the scenes and keeping objects and models updated that I'm thinking will take the brunt of my time. After all, loading and unloading is already included in the engine. I just have to make it do it the way I want it to. Keeping track of all the other data is another thing.

#5 Hmm, actually what I'm attempting to understand, is the actual dumping of the old.
I haven't really looked over how interiors are held in ram, but I believe that the textures are pretty much all loaded to a nice big linked list. I know if I need to change a current items texture on the fly, I reference the materiallist. So, how would you be dumping all the interiors at that end of the map, and their associated textures? Does streaming require you to maintain seperate lists of everything? And then you memset or whatnot a list thats no longer needed?

#6 Streaming is complicated. You will have to solve all those issues and more!

For the TSE terrain, I flag each terrainblock as being visible nearby, being visible far away, both, or not visible at all, based on its distance from the player (but not based on view frustrum, as that can change VERY rapidly). Then I load or unload data based on how the flags are changing.

#7 That's the reason I'm looking at loading/unloading via nodes (which is just the visual term I'm using; they would be lists in memory): managability. That way, I can compare, load, and flush the lists as needed. I need to look into how Torque handles garbage collection more, though, so I can see if I can use it or if I have to create my own handler. I'm thinking that because of the specifics of what I'm looking at, I'll have to either create a new one or overload the existing one. Each node is a list of interiors, models, entities, particles, scripts, triggers, and associated nodes (nodes that are directly accessible and therefore need to be cached).

I'm going to start with a node editor (based off of the world editor, but with a photoshop-esque layer GUI feel so that you can "show/load" or "hide/unload" nodes, and a DFX+/Shake flow interface to have a visual feel for how the nodes are linked) so that I can have a visual cue of whether the interiors are loading and unloading properly (since one of the reasons that I'm doing this is to be able to work with supposedly seamless environments, it should be transparent to the player), hopefully this weekend. It would simply allow you to load and organize your interiors, models, triggers, scripts, node joints, etc for each node much like the current world editor does, but rather than be an all-encompassing editor, be used for the editing of specific nodes or pieces of the game.

I'm going to sit down with the memory and resource management code tonight and sort through exactly what is going on so that I'm a bit more educated on where I need to start. That is if I don't get called into work.

@Ben*
Yeah. I know I'm in for one hell of an annoying ride. Luckily, I'm planning on using it for a single-player game so I don't have to worry about multiplayer restraints (though I would like to be aware of them). Which is a good thing since I'll have my hands full enough anyway. Having the system that I'm looking at seems so much more simple than dealing with terrain the way that you are (and they both have entirely different ends). Your way will help eliminate the annoying pop-up that LoD (or worse, streaming a patch of terrain with interiors and models suddenly appearing in the distance). My way, since it deals with small interior chunks is to avoid having to load an entire huge level. The design of the interiors and careful node connecting will be what eliminates pop-up in my implementation.

I'm still sketching out the idea and sorting through the source to see best how to implement it.

I can't wait to see your terrain implementation. I've heard nothing but good things from the hints that GG employees have been dropping!

*edit

#8 Ben, along with Stephen Z and David D I'm also working with the old terrain manager code. I'm keen to create code which I can use in common with the TSE-style terrain paging. Have you finalised the interfaces for the TSE terrain paging yet, and is it possible for us to start coding with them in mind even if the code isn't complete yet?

#9 Sorry, no dice. :)

#10 I am very interested to learn how to page terrain too.

I have been wondering if it would be possible to define a "world" as a grid of terrains, each of size nsquared (say 64k units) and have the "world" wrap around at the top, bottom and sides of the grid...

Jay

#11 @John: yes, it's certainly doable...Adellion has it working sweetly, and once I finish up my current milestone for my project, I'll be finishing up the patch against our project of the tpager system, and then patching THAT back against TGE 1.3.

When it gets to an alpha stage with most functionality present, it will be released back to the community.

#12 Sweet! I can't wait to see it. :)

Jay

#13 @John: Heh..don't hold your breath :) I hit a wall with the editors, so basically you cannot -do- anything with the terrain.

There are also some huge WAG's at the moment at the very low levels--rendering, collision, material maps, etc.

That being said, I have a grid of 16 (4x4) totally flat terrain squares, all with 4 big cones in the middle. Each grid square is roughly 1024x1024, for a total of a 4 mile x 4 mile (roughly) "continent".

It only crashes occasionally, but lighting the mission, and in some cases terrain paging can be signifigantly long (31-35 seconds for lighting, and I've seen 20 seconds of paging lag, which is -not- good enough).

#14 Stephen, I am looking into basically the same type of "paging" for my project. I am curious what your WAG is to what the paging lag of 20 seconds is comprised of?

Is this the time it takes to read your list of objects in the 9 sectors, load them, etc? Personally I am not concerned with Lighting and have almost counted it out because of the delay it would cause everytime a player shifted into a new "node/cell/whatever" of the 9 sectors.

How about the size, type and amount of objects? Do you feel this is what is causing your delay?

I have yet to start coding this portion of my project however, I am starting to work on the math etc.

#15 @punky: Honestly, I think it's partially the system I'm working with for development. The maps were completely empty, and it was only loading 3 new terrain tiles of approximately 500k each, so it's either a poor processing loop in my code, or poor system performance.

Once I'm able to step back into the tpager stuff (just before christmas most likely), I'll have a new dev platform and I'm hoping it will go away.

#16 How cool is this!?! I was just thinking I'd have to do this myself. I'm thinking of taking the RTS pack and then using terrain paging to do a large scale RPG, and this is perfectly awesome.

Rich

#17 @Rich: hehe...exactly what we are doing...exactly