No guts, no glory
by Gerald Fishel · 01/24/2009 (1:40 pm) · 13 comments
Since TGEA 1.8 was released I kind of gave up on some of the things I was hoping that would be taken care of, so I finally decided to eschew my inherent laziness and tackle them myself. And oh boy did I have no clue what I was getting into!
One of the things I've always lamented was the poor tools. So I've been spending a lot of time working on useful tools, because things need to get done. One of my major projects has been an art work-flow system for 3D Studio Max and Maya that allows my artists to build all kinds of goodies in Maya, and complete levels in 3D Studio Max, with one-click export/bake/optimize support.
I'm not quite ready to show any screenies from our game yet, but here is a quick shot with some of my glorious (cough) programmer art showing a few things:
Nothing too special there, but what this represents is a "level" exported from 3D Studio Max and loaded into the game engine without any additional scripting or configuration.
There are custom "Aegis" materials (Aegis is the codename for the project) which represent all of the various effects and features. What the artists see in 3D STudio Max is pretty much exactly what will be seen in the engine. I've extended the DirectX materials in Max so that they're able to read the HLSL files exactly how they are setup for TGEA instead of having to fool around with them to move from one to the other, which artists are singularly dangerous in so doing ;) This also includes post-processing effects like the glows that you see there, which I'll talk more about in a bit.
When a level is exported, any objects with materials that are flagged as lightmap receivers will automatically have lightmap UVs generated for them, and lightmaps and shadow maps will be baked. So there's no need for the artists to have to be bothered with adding UV channels for lightmaps, that's all handled by the exporter automatically.
The lighting model is a combination of static lightmaps, shadow maps, and some dynamic effects such as specular highlighting with up to 4 lights per material batch with shader model 2.0. (These are all custom renderable types, since the DTS format is far too archaic for me to mess with, and we're doing polysoup interiors rather than brush-based CSG so DIF is out.)
Additionally, the materials contain collision information, and other collision features can be specified with user data set for individual objects. On export the collidable objects are cooked with PhysX for either triangle mesh, convex data or collision primitives for loading in the engine. So for the most part the artists don't have to fiddle much with collision meshes. During prototyping they generally don't have to mess with it at all, but for final polish it's sometimes useful to be able to specify collision primitives, or convex meshes with up to 256 triangles which allows more physics possibilities since PhysX doesn't support trimesh against trimesh physics. These cases seem to be very rare though.
Which brings me to the first major headache I had. I was always unimpressed with the physics system in Torque, it was very difficult to do much of anything new with it. So I gutted it. Completely. All physics and collision related stuff is gone, and I replaced it with an abstracted physics framework, with rigid-body, soft-body, ragdoll physics, cloth, controllers that link to SceneObjects, etc. The "player" is controlled using character controllers, which can either be boxes, cylinders or capsules for smooth collision response/sliding.
The initial reference implementation is with PhysX but it should be fairly easy to create new implementations for other physics frameworks. If we ever decide to publish to a console we'll have to re-evaluate whether we want to spend an additional $50,000 for a license for PhysX on a console or use something else, and I figured I'd just make it as flexible as possible so if we have to change systems later it won't be too painful.
I've still got some work to do to get it syncing correctly between client and server, but I'm getting there. It's been painful, but very educational too ;)
The other major headache was a scriptable, chainable post-processing framework. So post-processing effects can be handled easily and chained together. The glow effect there in the screenie is one of the effects, which is completely independent from the built-in glow effect in TGEA. Motion blur, blooms and all other kinds of goodies are possible with it with relatively little effort.
----
In the coming days and weeks we'll be ready to start releasing more information about the game we've been sweating and bleeding over, and I'll be able to show some much better screenshots.
Once the game is done I'll probably be in a position to polish some of this stuff up more and offer it to other TGEA users. I'm not sure how that's going to work yet, and a lot of it will depend on what happens with T3D, but it's something that I'm thinking about.
Anyway, that's just a quick peek into some of the goodies I've been doing with Torque.
One of the things I've always lamented was the poor tools. So I've been spending a lot of time working on useful tools, because things need to get done. One of my major projects has been an art work-flow system for 3D Studio Max and Maya that allows my artists to build all kinds of goodies in Maya, and complete levels in 3D Studio Max, with one-click export/bake/optimize support.
I'm not quite ready to show any screenies from our game yet, but here is a quick shot with some of my glorious (cough) programmer art showing a few things:
Nothing too special there, but what this represents is a "level" exported from 3D Studio Max and loaded into the game engine without any additional scripting or configuration.
There are custom "Aegis" materials (Aegis is the codename for the project) which represent all of the various effects and features. What the artists see in 3D STudio Max is pretty much exactly what will be seen in the engine. I've extended the DirectX materials in Max so that they're able to read the HLSL files exactly how they are setup for TGEA instead of having to fool around with them to move from one to the other, which artists are singularly dangerous in so doing ;) This also includes post-processing effects like the glows that you see there, which I'll talk more about in a bit.
When a level is exported, any objects with materials that are flagged as lightmap receivers will automatically have lightmap UVs generated for them, and lightmaps and shadow maps will be baked. So there's no need for the artists to have to be bothered with adding UV channels for lightmaps, that's all handled by the exporter automatically.
The lighting model is a combination of static lightmaps, shadow maps, and some dynamic effects such as specular highlighting with up to 4 lights per material batch with shader model 2.0. (These are all custom renderable types, since the DTS format is far too archaic for me to mess with, and we're doing polysoup interiors rather than brush-based CSG so DIF is out.)
Additionally, the materials contain collision information, and other collision features can be specified with user data set for individual objects. On export the collidable objects are cooked with PhysX for either triangle mesh, convex data or collision primitives for loading in the engine. So for the most part the artists don't have to fiddle much with collision meshes. During prototyping they generally don't have to mess with it at all, but for final polish it's sometimes useful to be able to specify collision primitives, or convex meshes with up to 256 triangles which allows more physics possibilities since PhysX doesn't support trimesh against trimesh physics. These cases seem to be very rare though.
Which brings me to the first major headache I had. I was always unimpressed with the physics system in Torque, it was very difficult to do much of anything new with it. So I gutted it. Completely. All physics and collision related stuff is gone, and I replaced it with an abstracted physics framework, with rigid-body, soft-body, ragdoll physics, cloth, controllers that link to SceneObjects, etc. The "player" is controlled using character controllers, which can either be boxes, cylinders or capsules for smooth collision response/sliding.
The initial reference implementation is with PhysX but it should be fairly easy to create new implementations for other physics frameworks. If we ever decide to publish to a console we'll have to re-evaluate whether we want to spend an additional $50,000 for a license for PhysX on a console or use something else, and I figured I'd just make it as flexible as possible so if we have to change systems later it won't be too painful.
I've still got some work to do to get it syncing correctly between client and server, but I'm getting there. It's been painful, but very educational too ;)
The other major headache was a scriptable, chainable post-processing framework. So post-processing effects can be handled easily and chained together. The glow effect there in the screenie is one of the effects, which is completely independent from the built-in glow effect in TGEA. Motion blur, blooms and all other kinds of goodies are possible with it with relatively little effort.
----
In the coming days and weeks we'll be ready to start releasing more information about the game we've been sweating and bleeding over, and I'll be able to show some much better screenshots.
Once the game is done I'll probably be in a position to polish some of this stuff up more and offer it to other TGEA users. I'm not sure how that's going to work yet, and a lot of it will depend on what happens with T3D, but it's something that I'm thinking about.
Anyway, that's just a quick peek into some of the goodies I've been doing with Torque.
#2
you said cloth in your physics that there i would like to see. :P i know they might be working on t3d to where it can have cloth as well but this is good work your doing. keep up the good work and keep us posted.
01/24/2009 (2:35 pm)
wow that's allot of work to gutted the hole physics of torque and made your own.you said cloth in your physics that there i would like to see. :P i know they might be working on t3d to where it can have cloth as well but this is good work your doing. keep up the good work and keep us posted.
#3
If you are doing any sort of custom solution, why stick with DTS or DIF? Especially with more recent Torques that have more abstract rendering, as long as you can submit the right stuff to the render system, you're good. If it's animated - well, DTS does have a great animation system. But for static geometry...
It's easy - really easy compared to learning the internals of DTS or DIF - to write your own file format loaders. There are several good examples in the resources - like the 3DS loader that Matt Fairfax wrote years and years ago - as well as lots of code to learn from in the engine.
You can ALWAYS come up with a better solution for your specific situation than the GG guys will be able to, because they have to do things that work for everybody in every situation. Look at what Gerald did - he relies specifically on Max as his authoring environment and as a result his life is way simpler. GG can't do that - a lot of people will use Maya or other weird tools, and they can't very well expect every Torque licensee to also have Max.
The Collada support they've been working on has the potential to ease the pain here, but even there, it won't be as nice as a totally integrated solution would be.
I've never understood why people make deep mods to DIF or DTS when they could integrate their own pipeline with a tenth of the work... So my hat is off to you, Gerald, for making the smart choice! :)
01/24/2009 (2:43 pm)
You cut down overdraw by doing a z-prepass and maybe some occlusion culling or PVS work. If you are fill-rate bound you probably have complex shaders so the z-prepass will be a big win. Even on low-end cards, it's the shading that kills you, not the raw rasterization.If you are doing any sort of custom solution, why stick with DTS or DIF? Especially with more recent Torques that have more abstract rendering, as long as you can submit the right stuff to the render system, you're good. If it's animated - well, DTS does have a great animation system. But for static geometry...
It's easy - really easy compared to learning the internals of DTS or DIF - to write your own file format loaders. There are several good examples in the resources - like the 3DS loader that Matt Fairfax wrote years and years ago - as well as lots of code to learn from in the engine.
You can ALWAYS come up with a better solution for your specific situation than the GG guys will be able to, because they have to do things that work for everybody in every situation. Look at what Gerald did - he relies specifically on Max as his authoring environment and as a result his life is way simpler. GG can't do that - a lot of people will use Maya or other weird tools, and they can't very well expect every Torque licensee to also have Max.
The Collada support they've been working on has the potential to ease the pain here, but even there, it won't be as nice as a totally integrated solution would be.
I've never understood why people make deep mods to DIF or DTS when they could integrate their own pipeline with a tenth of the work... So my hat is off to you, Gerald, for making the smart choice! :)
#4
ICQ: 444836861 (Chat me up if you need a max model maker to be a guinea pig or part time help)
01/24/2009 (2:49 pm)
Sounds brilliant really.ICQ: 444836861 (Chat me up if you need a max model maker to be a guinea pig or part time help)
#5
01/24/2009 (2:51 pm)
@Adrian, there's nothing too special going on for managing the overdraw. Primarily I do some loose front-to-back sorting, some occlusion culling, and the interiors can be manually portalized where necessary. Using BSPs, cutting polygons, and crunching portal frustums actually costs us performance now, so I keep it pretty simple there; if a cell is visible through one of it's portals we basically just render everything in it.
#6
I had a quick try at figuring out the DTS format and the words that came to mind cannot be repeated on a family oriented website :D
01/24/2009 (3:11 pm)
@Ben, yeah... we still mostly use DTS for animated characters, though I gave serious thought to creating a new system there too, since it's lacking some of the "nice to haves", but won't really have time. Probably on the next project though.I had a quick try at figuring out the DTS format and the words that came to mind cannot be repeated on a family oriented website :D
#7
Hehehe.. I felt the same way when I tried to make the DTS loader work in a 64 bit pointer based environment. Pretty much gave up on it, I need to level up for that - and even then, I'll need to work on my fanatic woodchopper skill.
01/24/2009 (3:28 pm)
Quote:
I had a quick try at figuring out the DTS format and the words that came to mind cannot be repeated on a family oriented website :D
Hehehe.. I felt the same way when I tried to make the DTS loader work in a 64 bit pointer based environment. Pretty much gave up on it, I need to level up for that - and even then, I'll need to work on my fanatic woodchopper skill.
#8
01/24/2009 (3:32 pm)
Fantastic blog Gerald. I'm sure we'll be in touch soon re: how stuff like this works with Torque 3D. This is a very common, familiar workflow for professional studios, but does narrow the focus to particular tools as Ben said. There's some real merit to it and perhaps something very close to this is supportable in the long run in Torque 3D.
#9
01/24/2009 (3:59 pm)
Thanks, Brett. Yeah this does narrow the focus quite a bit. I've yet to come across a tool that does the kind of job that Max does for pre-rendering lighting and shadows though, so for us it's fine :D I ported the level exporter over to Maya too, but it turns out the light baking in Maya is terrible so my Maya guy just exports to FBX and brings into Max for final bake and export. 
#10
But to get back on topic, really looking forward to seeing the in game screenies that this is being used for.
01/24/2009 (4:15 pm)
@Ben, thanks for the pointer to the 3DS loader, wasn't aware of that. Just to make myself a bit clearer, I haven't attempted any mods, shallow or deep, to the DTS or DIF formats, I ended up creating my own simple text file format to load models in from, just to get an extra UV channel for lightmapping polysoup objects. And then tried using DIFs with portals to zone up the misssion area and drop polysoup interiors into the zones to cut down overdraw. Just my own spare time little R&D project, seems to work quite well, got some of it automated with some maxscripts, but very amateurish compared to this!But to get back on topic, really looking forward to seeing the in game screenies that this is being used for.
#11
And I agree, I'd _love_ to see what a real artist can do with Gerald's system!
01/24/2009 (4:17 pm)
@Adrian - No hard feelings, I hope, my comments weren't directed at you in particular. More general observations. :)And I agree, I'd _love_ to see what a real artist can do with Gerald's system!
#13
But that's just from my perspective in terms of being able to set up a level in my own hobby projects. For work related projects it might not be an issue.
01/24/2009 (10:19 pm)
Nice work Gerald! Personally, I'm not really keen on the whole "build everything in Max/Maya and export" sort of systems, since it basically locks out anyone that's not an artist from editing missions. I always figured the turn around for changing stuff would also be a pain.But that's just from my perspective in terms of being able to set up a level in my own hobby projects. For work related projects it might not be an issue.
Torque Owner Adrian W
OK now that I've composed myself...
This is exactly the kind of pipeline that I've been trying to inch towards with my Dts Merger and ATS file format, and using non-rendering DIFs to portalise up a polysoup level, but you are way ahead. I'd be interested to hear some detail about how you are cutting down overdraw, since it sounds like you are not using DIFs at all.