Game Development Community

Opening discussion on Terrain Paging

by David Dougher · in Torque Game Engine · 06/02/2004 (2:48 pm) · 56 replies

This thread will be used to discuss Terrain Paging.

First of all acknowledgements are in order.

J. Donavan Stanley did most of the updates to the Terrain Manager after its initial creation those changes became the project known as Gorpe.

Fredrik Svensson did a great deal of work on my first iterations of the Terrain Paging. Although the final version of the code is mine he provided a lot of useful feedback and helped with much of the hashing that goes into working out everything that was being touched and where.

Briefly then,
My Implementation is quite simple. It assumes a 3x3 tile system with the player at the center. As the player moves from one boundary to the next the appropriate tiles are loaded and the tiles furthest from the player are dropped.
Worst case scenario is moving at a diagonal from corner tile to corner tile. This causes the loss of five tiles and the loading of five.

A couple of simple checks are run to determine which tile the player is on and entering a new tile initiates a page check. Depending on which direction you are moving the further tiles are let go. The the new tiles are loaded. There are special tests for edge conditions and corners.

Essentially, if you run to an edge or a corner of the world the tiles you currently have are retained and you move off of the center tile. The reason for this is simplicity - it is easier to maintain the 3x3 matrix then to special case all the differences. Also, if you run into a corner you will still have to come back out -- meaning that we have already cached the tiles you will run to eventually when leaving.

As you return from a corner or edge we move you back the center tile.

About the author

Owner - Pariah Games, Adjunct Professor - Bristol Community College, Mentor - Game Design - Met School Newport, Mentor - Game Design - Met School Providence

Page «Previous 1 2 3 Last »
#1
06/02/2004 (3:13 pm)
There is a hidden assumption in there by the way (that is a pretty good one for most worlds, but not all)--your third dimension is fractional--in other words, you can't move "down" or "up" substantially enough to go to a new tile.

Our design uses the exact same concept of a 3x3 "currently loaded" configuration--we call it a "9grid" for easy terminology. We recently recognized this hidden assumption when considering how to have subterranean tiles, and I'm not positive yet it really is an issue, but it might could use some brainpower.
#2
06/03/2004 (7:33 am)
@Stephen,
how would you create a "sky" or surround for a subterranean tile if it's a terrain rather than an interior or shape. I like the concept.

@David,
On the surface I liked the extensibility that J.Donovan was planning (XML, Python, etc). I'm just not sure what the implications are in the long run. In addition to paging, I think these GORPE extensions should be discussed up front. P.S. TSE EA (order capability) is available!!!!!
#3
06/03/2004 (8:54 am)
@Desmond: I don't know! Currently, we know that our game will have "underground" regions, some very large. We also want to have the ability for tunneling and excavation, and taking those thoughts to their logical conclusion made me realize that the terrain paging (not just this design, we were working on it our selves and pretty much had the same design as David, just not -nearly- as much development) doesn't really take into account the ability to go "down" and load a new terrain tile.

I'm not even sure yet if it's required, or even smart, to include it. It may be as simple as having a portal mechanism to a separate "terrain tile area". That doesn't seem elegant to me however, but then again, having to page in a 3x3x3 "grid" doesn't look elegant either.

Edit: It may be an obvious solution as well to simply have all underground areas treated as interiors, and for these particular areas, treat them as single tiles. Again, our design is still exploring the idea.
#4
06/03/2004 (11:32 am)
@Desmond Been reviewing TSE all day. Got it last night and started reading then... Wife dragged me off at 2AM.

@Stephen Not sure about what you would try and do here. Seems to me that for anything below ground you would switch over to something like Quark or Worldcraft for below ground areas or "zone" if it is beyond what you want those tools to do.

Another issue will be opening holes in terrain for buildings and such using absolute coordinates. If I remember, currently only the last tile loaded has holes opened for it correctly by the Terrain Manager.

A 3x3x3 grid won't solve your porblem since you should be able to go down any arbitrary amount - not a specific plane distance - not to mention the hassle of distinguishing between a plane under a mountain and a nearby valley which could be at or below the height under the mountain.
#5
06/03/2004 (5:05 pm)
My little Latitude C840 with GF4 440GO card is having problems with TSE; been trying to see if there are any DELL upgrade drivers to no avail. I have two other systems with GF3 and GF4 but I sure like to kick back with the laptop.
#6
06/03/2004 (5:45 pm)
Heya Des,

I have the same issue with my Inspiron 8200 and its cruddy GeForce4 440 Go :( damned thing aint not Gefore4 thats for sure!! Think JoshW or a community menber said the 440 Go is less than a Gefore2 ...
#7
06/04/2004 (2:19 pm)
Yeah, Thanks DELL!!! I went to the Dell customer forums yesterday and came across a guy that has an inspiron 8200 and he was pointing to a site that has omega drivers that he says brings it up a bit. Wasn't sure I was willing to try them; but after trying some other stuff yesterday and totally messing up my video drivers, I discovered the WinXP RESTORE function...what a life saver. Restored back to an hour earlier and back up and running. So I might try out that Omega driver and let you know how it goes; Dell sure doesn't seem to be moving fast enough.
#8
06/04/2004 (4:35 pm)
Guys, with all due respect, can we leave the TSE discussions in the other forums? This thread is about the Terrain Manager, nearly as I can tell from a design/expansion design perspective.
#9
06/04/2004 (7:49 pm)
Yeah, true that. I'm all for TSE exuberance but on-topic conversation is for the ages. :)
#10
06/06/2004 (10:57 pm)
Heres an idea we're working on, also based on 3x3 tiling:

We're writing a world editor outside of torque using DirectX and fractal landscape in order to more easily create an entire world.

The landscape will be fractally generated using Perlin Noise, procedurally altered and masked by another fractal to get variation in landscape (instead of endless mountains or endless hills).

The world editor will not, in fact, save the actual height data. Rather, it will save the seed values for the fractal and co-ordinates/properties of objects placed on the landscape, such as houses, path nodes etc.

In other words the editor is more of a world explorer that allows you to specify where objects are in the big world while seeing the shape of the underlying landscape.

Local custom deformations will be possible by placing "patches" over the landscape and deforming them. These will be both simple procedural patches (like flatten/smooth) and bezier spline grids that can be deformed. Either will deform the underlying landscape locally. This will allow you to flattern out that little bit more to fit a coastal town, or whatever.

The fractal generation code and the various other things we have planned will be written in C++ using a plugin architecture we've developed so others can eventually add features. Also it means we have the C++ code ready to replicate what the editor is doing in the game engine.

General information such as seed values, fractal properties et al for the particular world (continent) will be stored in a ".MAP" file, as will information about supporting files such as patch files.

Objects (ids of torque objects) and their properties will not be stored in the ".MIS" file but a random access file that functions as a simple database.

In Torque, we'll use the 9x9 logic described in posts above with a twist. The game loads the ".map" at the start to get a world seed and so on and when it comes time to load the terrainblock, instead it just uses our c++ code to generate the appropriate local perlin noise landscape, based on x,y. So it doesn't really load the landscape, it calculates it. It seems quite easy to alter the landscape loading function. We've managed, for instance to get it to load height values from a RAW file instead of a TER file.

Where is it going to get textures you may ask. Well, that will also be defined in our map editor. All texturing will be procedural with fractal distribution and that information will also be in the ".MAP" file, along with any "texture patches" for custom local texture.

Because raw heightfield data takes less resources than a terrainblock, we'll be calculating the heightfield for a 5x5 rather than a 3 by 3 block area, meaning that when time comes to load terrainblocks, the data will already be in memory and just have to be transferred to the mesh.

Anyone who's worked with complex overlayed procedurally altered fractals will know calculating 5x5's worth of height data will be a huge hit, but here's the beauty. When you first appear it will pause to do the 5x5 terrainblock heightmap.

After that every time you move one terrainblock point in any direction it will just shift all the height values in the array in the opposite direction and calculate 1 line of height values. The absolute minimum resolution of progressive loading.

Bear in mind these are the height values we're getting the terrainblocks from, not the blocks themselves. So we've always got the height data for a 5x5 grid around you with you in the center. From this we can easily and quickly load terrainblocks to have the correct 3x3 grid as you move over boundaries.
#11
06/06/2004 (10:57 pm)
Objects will not be loaded from the mission file but rather from the custom "database" according to distance etc. Not sure how this will affect various physics functions but we're not planning on using baked in shadows for statics and will load by distance and size. We may decide to do it by terrainblock if that's not feasible but I like the idea of objects that are a block away and huge loading but small objects over the next hill not loading.

There's lots of other stuff we've thought of but that's the basics. Right now its a great plan but we're only as far as a sophisticated plugin system for our world editor, some Perlin noise code and a preliminary implementation of a dynamically deforming DirectX mesh for the world editor.

Waddya think?
#12
06/07/2004 (3:42 am)
Let me preface this by saying I'm a beginner programmer and Torquehead, so I really have no idea what is and is not possible with the engine yet.

I'm wondering though why folks are beating themselves up to create grids? For most games, the current setup would be fine. The two exceptions I could see would be MMORPG's and Flight sims. And in that case, I would think it would be nicer to just have one large terrain. Not grids.

I'm thinking along the way that Asheron's Call does it. Instead of the game engine needing to track each player and 9+ "grids" that surround them it's a simple X.Y for each player on one large grid.

I would think if you are going to reprogram the way TGE/TSE handles terrain, that would work the best, but then again, I really can't speak for how difficult either way is to code.
#13
06/07/2004 (4:18 am)
@Chris, a couple things. First after chatting with the folks from Turbine for the past few years I can tell you that the way Asheron's Call works is not nearly that simple. In addition, it has some rather visual popping effects at distances as people pass thru different server zones.

AC has multiple handoff servers working as a cluster with dynamic load shifting. Multiple servers are able to assist each other. In addition, AC had to position cities in specific locations so that different quadrants of the the same city could be handled by different servers if the load went up. Visually they are doing something similar to what we are talking about on the client side. They are simply using smaller regions or patches - not the larger tiles we are currently swapping.

We could achieve much the same effect if we reduced each single terrain tile to about nine tiles and then used the same algorithms to page. It would be faster but we would have significantly more tiles and there would be issues with waterblocks and there could be visual popping at distances.

This would be possible with Farren's idea above, or by simply changing the way the terrain tiles are written out to break each tile into 9 smaller tiles when written. On the client side you would read the tile pieces as you move.
#14
06/07/2004 (4:35 am)
@David Dougher

As far as I understand, this will only help the client reducing load and not the server? Right?
#15
06/07/2004 (5:13 am)
Yes, unless you split the servers this will only reduce the load on the client. There is a slight performance hit for the client in that with multiple servers you have overhead for swapping from server to server, but the performance hit is generally lost in the long lead time of the servers exchanging data about the client before it transitions from one system to the other. It only shows up under extreme load conditions and by then you have probably seen it much more severlely in other systems.

However, you also need to remember that with all the load on a single server there are also performance gains and optimizations. The server now does not handle any of the grpahics load - that is left to the client. Only collision boxes are handled server side.

There is a large overhead associated with handoff from one server to another - not just characters, but also projectiles and their associated vectors and issues of things like trading objects when one character is on one server and the other is on a different server (a major source for duping cheats.) Which is why a couple of recent MMO's have opted for a single "big server" solution.

But you are correct client side paging essentially benefits the client only not the server.
#16
06/07/2004 (5:22 am)
Farren,

Terrain deformation in real-time is a cool idea but a nightmare to deal with in an MMO. The problem is not deforming the terrain which can be done relatively easily with the terrain algorithm chosen by GG, but with the updates that must be generated to all players in the game world.

Essentially you would need to maintain a list of all terrain deformations and their locations and check constantly as each player enters a new patch of territory to see if that character needs to be streamed a new terrain patch. Then you need to light the terrain patch properly. Such a list could grow to unmanageable sizes rapidly - especially in a war situation where you could see explosions upon explosions.

If you are only talking about terrain shaping before the game is done then I would ask what you found wrong with all the terrain deformation tools in the current torque editor? Mark did an outstanding job rewriting the tool. It is currently limited to a single terrain tile but I would seriously consider using the terrain editing tools in torque and extend them to multiple terrain tiles rather than try and reinvent them. If you really feel the need for a seperate editor at least consider using the same functions already in torque - you bought the code. You couldn't sell the editor separately - except to a registered torque owner, but you couldn't do that anyway without violating the EULA since you are dealing with torque's underlying codebase. You would gain a consistency in tile generation between what torque generates and what you generate.
#17
06/07/2004 (5:51 am)
@David - I see your point. I _thought_ AC used a seemless terrain, but that's not the case. Though another thought popped into my head. Other than the known water edge bug, is there any other issues with just increasing the terrainblock size up to it's max. That would greatly increase the you could have in one "grid". Though I'm guessing it impacts the visual quality of the terrain?

Speaking of which. Has anyone ever calculated out the "real world" sizes of a grid based on the various terrain block sizes? I believe I read that using 8 gives you a 2KM square grid, but I can't be sure of that.
#18
06/07/2004 (6:01 am)
Quote:Terrain deformation in real-time is a cool idea but a nightmare to deal with in an MMO. The problem is not deforming the terrain which can be done relatively easily with the terrain algorithm chosen by GG, but with the updates that must be generated to all players in the game world.

We'll be using a relatively low occurence form of terrain deformation--players will have the ability to flatten out a specific section for building placement (making a foundation, as it were). Our plan is to store these as a terrain overlay and trasmit to the client on grid paging, and handle any "as you watch" situations with a small overlay update.

Based on performance metrics, these overlays will become "semi-permanent" and added directly to the terrain grid over time, and the new terrain grid will be transmitted to the player as part of their grid "library" during initial patch/update for the next login session. So in effect, when it becomes too much bandwidth to transmit an overlay each grid page, we'll apply the overlay directly on the server, and then make the specific modified grid available next login.
#19
06/07/2004 (6:17 am)
@Stephen,

A good system, and one which we also looked at. However, how are you planning on dealing with players "digging" under the corners of buildings, trees, bodies of water, and other semi-permanent structures? Were you going to limit where players can dig or flatten?

Also the issue becomes one of customer satisfaction at some point as well. If people can dig or flattten they will want to be able to do it everywhere. They will complain when they can do it "here" and not "there". They will want to know why an explosion will not make a hole in a war or not make a hole in a wall, but a farmer can make a hole with a shovel.

This is not to discourage you. In fact, if you solve it I'm sure we would all really like to see the code! Resource! Resource! ;) Just want to make sure you are planning for all sides of a very thorny issue.


@Chris,
The standard 8 size grid is 2km per side (about 1 sq. mi.) If you check around the forums a bit you will find discussions in which all the information has been broken down.

The waterblock has a known issues in that, at terrain sizes above 8, the waterblocks will develop holes in relationship to the terrain. There are issues with two waterblocks being placed adjacent to each other (don't know if that was ever solved), and a third with the waterblocks penetrating below its stated depth - i.e players passing under a water block in a tunnel can slow down or drown.

Some of these issues have some reported fixes, but all of the fixes reported to date have not worked in all cases from my readings. The issues may be resolvable but to date no one has reported a complete success.
#20
06/07/2004 (7:13 am)
@David:
Quote:A good system, and one which we also looked at. However, how are you planning on dealing with players "digging" under the corners of buildings, trees, bodies of water, and other semi-permanent structures? Were you going to limit where players can dig or flatten?

Also the issue becomes one of customer satisfaction at some point as well. If people can dig or flattten they will want to be able to do it everywhere.

Initially, the design was just to avoid terrain problems when placing buildings as a player. We're exploring the ideas of "freeform" digging, especially for things like moats, and siege tunnels. In fact, we would expressly want folks to be able to "dig under towers/walls" and have functionality to cause structural damage (that's a lot of physics, we know, hehe). We'd like to have free-form terraforming, but as you mentioned, there are performance issues, as well as unexpected impact on the environment, and function creep ("ok, I just dug a moat, and I even dug a trench over to that lake--why isn't my moat filled with water?").

Quote:This is not to discourage you. In fact, if you solve it I'm sure we would all really like to see the code! Resource! Resource! ;) Just want to make sure you are planning for all sides of a very thorny issue.
Our current implementation tasks are well behind this level of the design (hell, we haven't even patched in the Terrain Manager yet), but when we get to that point we'll definitely bring it back to the community.
Page «Previous 1 2 3 Last »