Torque pipeline progress

If you look at the current art pipeline for torque, you can see that there are some issues in terms of usability and process that could do with getting some attention.

The main problems as I see it are:

> The reliance on "exporters" for DCC apps is too much for GG to maintain
> The scene structure implies far too much within the modelling app
> The reliance on a non standard format means that there are issues when working with third parties
> The extensibility of the exporter based approach is in question as new features become available
> There are obvious flaws and missing features in the current pipeline
> The pipeline itself isnt readily extensible with third party tools

Now, I'm not saying that you cant produce great artwork for torque, as clearly you can. But if you start trying to use it from anything but a professional's standpoint, clearly it becomes troublesome. Specifically, for less experienced developers I feel that the process of using thier creations is too complex.

So I've started addressing these and other issues, which is what I'm going to write about here. I wanted to throw out my plans in order for people to feed back and possibly spot potential issues I've not considered.

First, let me outline the basic methodology.

I'm aiming to create a flexible pipeline, that will allow content production in as many applications as possible. By moving to an "import" based methodology rather than an "export" based one, I hope to leverage the fact that file formats change far less frequenctly than DCC apps, plus also capturing many more of these applications in the process.

So where to start?

The ideal, is to allow the user to simply select ANY supported model format (formats I'll enumerate later) from within the world editor and use it as a shape. I plan on having this functionality, but I feel there is some benefit to a slight different strategy initially.

My thinking, is that before I start adding format support directly into torque, I will get a feel for the formats and add some much required functionality into the pipeline beforehand. So my first goal, will be to create a pipeline that allows for conversion from external formats to internal formats. So in a nutshell, you will be able to go from an input format, to an internal format. Lets enumerate them now:

External input formats:
3ds, collada (dae),fbx,obj,mdl,md3,?????
Internal output formats:
dts,dtx

You might have noticed that DTX at the end. Thats going to be an interim test format, so that I can incorporate added functionality into the engine core side of things (i.e. add UV layers etc) while still allowing legacy support for dts's.

So, how does all this work?

Here's a diagram that should give you an overall picture of the structure of this application.

Click here for higher res version
The important parts are the importer, the internal scenegraph, the conditioner and the exporter. These are all scriptable by design, such that more complex control is possible.

So what do the parts do?

Importer

This is the interface for loading a given file format into a scene. Essentially it converts an external format into a format that I can easily understand and document, its only purpose is to read from the file and stuff data into the scene.

Scene

The scene is the data repository. There is a manager interface because some situations will require multiple individual scenes to be present (for example, you might want seperate files for each LOD and to use a conditioner to then create a single scene from those).

Conditioner

I coined this term from the Collada docs. Basically, its a process that somehow either processes the data, or transforms it. There are so many different things you would use a conditioner for. The prime one I'm interested in, is the ability to call third party apps for baking ambient occlusion or GI lighting. So for instance, you would write a gelato conditioner that would write out an interim scene and then use NVidia's Gelato renderer to bake out individual lightmaps for each object. A simpler conditioner would perhaps reduce the polygon count on a given mesh, or even simpler still, automatically compute the bounds box of a given shape.

Exporter

This is the final peice of the puzzle. You somehow have to use this data in torque. At some point, the internals of this app will be incorporates into the main engine such that this process isnt required. But for now, it will simply take a given scene and write it out to either dts format, or the new extended capability dtx format.

The main thing is that this is all extensible. Primarily, I can see two main efforts here. Beyond actually reading the given formats. The first is to create a set of useful conditioners. The second is to make the whole process a lot more user friendly.

The second part of that, will require the creation of a "tool" application. Right now I'm thinking that the initial release of this application will be command line based. I understand thats counter to what I say about ease of use. But I intend to remedy that with a follow up UI application that takes the core code and allows you to perform the same operations only graphically. If you've used any of the high end DCC apps in command line mode, you'll see where I'm going with that.

So as you can see, its a considerable task, although to be fair, its not as complex as it sounds. Its basically reading and writing data for the most part. I'll post some more complete design notes later and hopefully get some base code fleshed out and documented enough to show how it all works. One thing I'm very aware of is not making this whole thing more complex than it needs to be. Having working defaults for everything where possible and generally making an easy to use and modify api is important.

So, thats what I'm up to. What about you? :)