T3D Terrain Pager and FlightGear Data Source, v2.0

So... for anyone who's wondering what the heck happened with my PhysX branch after the initial burst of energy, well, this blog is not about that, but never fear! That project is sitting right in the path of my primary work, so it will definitely be getting pushed forward in the not too distant future.

However, this blog is about something else. I've had a back burner project on my mind for the last couple of years, but this winter it was finally time to give it a hard push forward. Readers who've been around here for a while may remember this blog of mine.

To make a long story short: My primary interest in game technology has always been more in the direction of real world modeling and simulation than pure entertainment. My bachelor's degree is actually in geography, rather than programming, and it turns out most of the things I really want to do with a game engine just don't make sense without a world to put them in.

In short, unlike most game developers, I'm really not very interested in fictional landscapes. Also, being not much of an artist, I'm far more excited by procedural solutions and/or taking advantage of existing work than I am by the concept of sitting down in the terrain editor and pushing up hills.

Enter: FLIGHTGEAR!

There is no time to explain it in detail, you can find all the information you want at flightgear.org, but for our purposes suffice it to say that FlightGear is a long standing, very successful open source flight simulator project. If you're interested in flying virtual airplanes, by all means download it and check it out.

However, I'm not interested in flying airplanes at the moment. What brought me to flightgear are two salient facts:

1. over the years their community has established a complete 3D model of the entire Earth, albeit a very low resolution one, and
2. as part of the process, they have created a very useful set of tools for improving this world model, using GIS data.

Now, because this is already way too many words in a row, here's a picture:



Without further explanation, your first reaction is probably "OMG, what an incredibly ugly world you're showing me, why are you wasting my time?"

But before you give up on me, understand that what you're going to be seeing in this blog is a "before picture". This is a proof of concept demonstrating a barebones functional connection between FlightGear and Torque, with absolutely no steps taken toward making it prettier.

Anyway, the core of what I'd like to share with you all are two things:

1. A new terrain paging system for Torque.
2. Two new networking classes, dataSource and worldDataSource.

First, the terrain pager. This project has gone through two major phases before it got here. The first was called "T3DTerrainMaster", and it was my first attempt at paging terrain tiles in Torque. The second was called FlightGearWorldServer, and it was pretty much like what I'm describing here, except it was mostly geared toward Unity (sorry, I strayed away for a couple of years!) and the code for it was mostly a steaming pile of sh*t.

Which brings me to what's new here: I've taken the last several weeks and painstakingly dissected my previous code for this project, reoriented it toward T3D again, and most importantly, broken it apart into the most portable and modular chunks possible.

What I came out with is a total of six code files plus a few modifications elsewhere. The terrain pager lives in terrain/terrPager.cpp/.h, and it works by flipping through a grid of the same sized tiles as the player moves around the map. Any legal size for a terrain tile will work, and you tell it the properties you need by seeding the system with a single base terrain tile of your choosing. For my purposes I am currently using terrains with a 256x256 heightmap resolution and a square size of ten meters. I realize this is pretty coarse for a first person game, but like I said, it's a first pass. Eventually I'd like to make it deal with tiles of the same overall size but different resolutions, to allow some areas on the map to be high detail while others are not... but later.

The terrain pager uses a simple method of loading tiles via a "load radius" entered by the user. I had some trouble dropping them cleanly, so for now I just keep everything loaded until you shut down the game, but eventually the tiles will be dropped when they exceed the "drop radius".

The system is geocentric in the sense that it requires a latitude and longitude for map center, and then it names all the tiles based on the lat/long of their southwest corner. It would work just as well for a fictional game terrain, but you would have to feed it your own terrains of course, and pick a fictional lat/long for map center.




For simple terrain paging, all you need is a collection of appropriately named .ter files, but of course the heart of this project is the connection to an external data source for this information. This leads me to the other part of the project, which is located in sim/dataSource/, and consists of the base dataSource class and a worldDataSource class which inherits from it.

The purpose of these classes was to have a very lightweight and simple networking interface between Torque and another application. I know that Torque has its own networking classes, obviously, but I wanted something geared toward sending a series of requests to another app through a socket. For this I did not need the extreme efficiency, or complexity, of Torque's networking, but I did need something that would be easy to deal with on both ends.

So, I wrote the dataSource class in pure, portable C++. The idea for it is that both the client and the server app can compile the exact same code, and have access to methods for either listening or sending data, and receiving or requesting connections. The dataSource class by itself is not abstract, it can be run by itself and establish a connection, with a base communication request that basically does nothing but shake hands and hand over its current tick, just so we can tell it's working.

The worldDataSource class, then, inherits this basic functionality and adds a bunch of methods for requesting specific world data. At the moment this is limited to 3D terrain data and skybox images, but in the future it could easily include weather information or many other types of data. The worldDataSource code is as identical as possible between T3D and FlightGear, with the only variation being things like String data types, Con::printf vs printf, and then the actual nitty gritty work of either serving up or receiving data.

The point of separating dataSource from worldDataSource is to make it easy to inherit other classes based on dataSource, in order to use the same code for other applications serving up different kinds of data.

Communication is done through 1024 byte packets, which are broken down into the following simple protocol. For simplicity, all numbers are handled as four byte floats and then converted back to integers if necessary.

* first four bytes is a number representing total count of requests in this packet.
* after this, all other data consists of specific requests, each of which starts with a single numeric code representing which type of request this is (initialization, terrain request, skybox request, etc.) and then the rest of the data is a series of numeric arguments, the number of which is predetermined by request type.
* there will soon be a way to handle character strings, ie paths, after the numeric data, but not today.

That's pretty much all there is to it at this point. In the TerrainPager.mis file, there is a TerrainPager object defined, and this is where you give it your mapCenter coordinates, your desired skybox image resolution, and your desired tile load/drop radii. Also, this is where you define whether or not you want to connect to a dataSource.

In terms of end uses for this tech, it is quite easy to for the developer to generate a large number of terrain tiles and ship them with a project. The skyboxes are quite a bit more difficult to handle this way, however. When running a live data source, the skybox gets replaced every couple of minutes, and hence it stays pretty current. However, I've experimented with saving them to a grid as well, so you could refresh them without the data source running, but had limited success. This eats up disk space pretty fast, and it's pretty difficult to avoid very unrealistic skyboxes from many vantage points, especially on rough terrain.

Ultimately, my goal is to do a lot of work on the flightgear end in terms of making it more invisible and less expensive, so as to be able to run it alongside without being as distracting as it is now, but for the moment, I'm just happy to have this much working at all.

With the barebones system in place, the way forward involves many improvements. First, the terrain you see here from FlightGear is 90m resolution, hence the giant flat polygons everywhere. Like I said, this is the "before picture", but I already have data in hand for ten meter resolution coverage of the same area, which looks MUCH better.

I would also very much like to apply some spline curve or other logic to fill in additional data between the known terrain points, to make a torque terrain at perhaps 2.5m resolution with curves filled out.

The many strange spikes and artifacts you see in the city are the result of my naive use of a getHeight function in FlightGear which fails to distinguish between terrain and added buildings and models, so that will go away soon I'm sure.

The very rough map of Eugene, filled in by a single texture, is just vanilla unimproved FlightGear map data, which can be replaced by an actual street grid and much more detailed shapefile data from OpenStreetMap.org and other sources.

Of course, being in Torque, once we have a street grid made up of textures, it will be just another step to harvest vector data out of this and create decal or mesh roads where there should be streets, and then create T3D rivers and streams, or water blocks, where there should be water.

Forests and ground covers can be added based on material types received from FlightGear, and of course terrain detail textures would improve matters substantially.




This blog is already too long, so I'm going to hold back on my personal plans for this technology, but suffice it to say I developed it... for reasons. However, if your curiosity is piqued at this point, be assured that everything I've described so far is freely released into the public domain (my apologies if I haven't made that clear in the file headers yet, next update!)

First, the playable demo, ie the game directory with an executable included, is here:

github.com/ChrisCalef/terrainPager_game

The source code is in my Torque3D fork, under the terrainPager branch:

github.com/ChrisCalef/Torque3D/tree/terrain_pager

The first link will be enough to let you test the terrainPager and run around a few pre-generated terrains. If you want to run the whole process with flightgear and experiment with generating your own terrains, then you'll have quite a bit more downloading to do and I can't guarantee everything's going to work for you yet, but get in touch with me if you're really interested and I'll try to help.

My modified version of FlightGear can be found in this repository:

github.com/ChrisCalef/FlightGear_dataSource

However, the one part of the project that totally is not done yet at all is dealing with the massive amount of data you need to run FlightGear. To make this project friendly and portable I need to prune out everything that is not needed for my purposes, but without detailed knowledge this is much easier said than done. In the meantime, very interested people can go straight to the source and take on the whole download:

gitorious.org/fg/fgdata

However, be warned, it's multiple gigs. There is a version floating around without the massive airplanes directory, but like I said, contact me if you're really interested, otherwise I'm going to assume this is just a show and tell session for now.

And now, I must get back to more pressing concerns. I hope this blog has been at least somewhat entertaining for someone.