It's Teh Economy Stoopid
by Steve Acaster · 04/06/2009 (3:15 pm) · 31 comments
Actually, it's the lightmaps. Constructor crashes on export? Bets are it's the lightmaps. Funny dark cuts along the edges of the brushes visible from certain angles and certain distances? Not brush borders, but lightmap borders.
Why does this all happen? Because geometry scale is 32, small scale lightmaps (like 8) cause trauma on larger brushes when calculating the lightmap itself. Keep brushes smaller for less lightmap calculation map crash trauma. Compressed edges of lightmaps visible? Make lightmap scale the same as geometry to dispell (though to be honest you'll probably still see a few of the little critters lurking out in the far distance if you have a long, long surface like a road). There's also an occaisonal light bleed in some interiors with this.
What the world looks like immediately after it spawns from my id
So, with that finally realised, we set out to design Map2. Map1 (aka High Wold, it's the urban pics in previous blogs) was created as a large test DIF using large facade blocks as the buildings. Rather urbanized, terrace streets. Map2 was designed for more individual buildings, detached, semi-detached housing. The idea being to create a series of prefabs which I can then drop into editor/modeling program, arrange and then export en mass as a single object (DIF or DTS).
And it all seemed to go okay like that. I did attempt a new method of creating a mesh model in Blender and then exporting it as a mapfile, but Blender's map exporter isn't up to anything particularly complicated, and the end result looked like a mangled, disjointed mess on close inspection. Blender can export mapfiles fine - if you stick to the basic rules of being BSP friendly (build it using individual blocks - think lego).
Church tower is 400m away, just out of rifle range but not sniper or lmg
So, establishing that prefabbing really is the way to go, and with a DIF version knocked up, I recreated the whole thing as a DTS. Whilst polysoup collision has been available for a while, the lack of lightmaps has meant that it just didn't cast realistic shadows onto itself or other meshes. This can look a bit odd to say the least. However, there is new tech on the horizon which may well sort this problem out, and which includes scandulous mention (it's not really scandulous, I'm just using a random adjective for teh interwebz draaama) that DIF will be more performance friendly. With my Map2 copied into both DIF and Polysouped DTS, it'll be interesting to see what the difference is in-game using said new tech.
BSP version, roofs are NULLed
To be honest I was having so much fun building the DTS version that I imbellished it quite a bit, so it's got 7k faces instead of 5k as the DIF version. Performance-wise in current tech (TGEA1.7.1 'cos I haven't transfered all of my custom scripts over to 1.8.1 yet - though from what I've experimented with - the High Wold map, 1.8.1 gives noticeably better performance than 1.7.1) the DTS polysoup wins out with an FPS of 30% higher than DIF (based on 5 locations and views at 1280x768 resolution with 1700m visibility). That's an max-average fps of DIF 125 and DTS Polysoup 165.
Polysouped DTS version - spot the difference!
Apart from the DTS lighting issue there are also a couple of other problems, such as the model vanishing when the fog/maxvis hits the centre of the bounding box (problematic with large meshes in low visibility environments - but fine with a long maxvis like I'm currently using). I'm a great believer that in making large environments as a single object is better for performance than making hundreds of individual objects and then sticking them in-world together. No matter how many surfaces it has, it is still just one object for the processor to initialize and then worry about, instead of 50. Another issue I've noticed with current tech's polysoup is that the colour/tone/intensity of the textures aren't as vibrant as they are when applied to DIF. I expect this is down to the lightmaps.
Anyhow, it'll be interesting to see what the new tech brings. Roll on Beta.
Why does this all happen? Because geometry scale is 32, small scale lightmaps (like 8) cause trauma on larger brushes when calculating the lightmap itself. Keep brushes smaller for less lightmap calculation map crash trauma. Compressed edges of lightmaps visible? Make lightmap scale the same as geometry to dispell (though to be honest you'll probably still see a few of the little critters lurking out in the far distance if you have a long, long surface like a road). There's also an occaisonal light bleed in some interiors with this.
What the world looks like immediately after it spawns from my id
So, with that finally realised, we set out to design Map2. Map1 (aka High Wold, it's the urban pics in previous blogs) was created as a large test DIF using large facade blocks as the buildings. Rather urbanized, terrace streets. Map2 was designed for more individual buildings, detached, semi-detached housing. The idea being to create a series of prefabs which I can then drop into editor/modeling program, arrange and then export en mass as a single object (DIF or DTS).
And it all seemed to go okay like that. I did attempt a new method of creating a mesh model in Blender and then exporting it as a mapfile, but Blender's map exporter isn't up to anything particularly complicated, and the end result looked like a mangled, disjointed mess on close inspection. Blender can export mapfiles fine - if you stick to the basic rules of being BSP friendly (build it using individual blocks - think lego).
Church tower is 400m away, just out of rifle range but not sniper or lmg
So, establishing that prefabbing really is the way to go, and with a DIF version knocked up, I recreated the whole thing as a DTS. Whilst polysoup collision has been available for a while, the lack of lightmaps has meant that it just didn't cast realistic shadows onto itself or other meshes. This can look a bit odd to say the least. However, there is new tech on the horizon which may well sort this problem out, and which includes scandulous mention (it's not really scandulous, I'm just using a random adjective for teh interwebz draaama) that DIF will be more performance friendly. With my Map2 copied into both DIF and Polysouped DTS, it'll be interesting to see what the difference is in-game using said new tech.
BSP version, roofs are NULLed
To be honest I was having so much fun building the DTS version that I imbellished it quite a bit, so it's got 7k faces instead of 5k as the DIF version. Performance-wise in current tech (TGEA1.7.1 'cos I haven't transfered all of my custom scripts over to 1.8.1 yet - though from what I've experimented with - the High Wold map, 1.8.1 gives noticeably better performance than 1.7.1) the DTS polysoup wins out with an FPS of 30% higher than DIF (based on 5 locations and views at 1280x768 resolution with 1700m visibility). That's an max-average fps of DIF 125 and DTS Polysoup 165.
Polysouped DTS version - spot the difference!
Apart from the DTS lighting issue there are also a couple of other problems, such as the model vanishing when the fog/maxvis hits the centre of the bounding box (problematic with large meshes in low visibility environments - but fine with a long maxvis like I'm currently using). I'm a great believer that in making large environments as a single object is better for performance than making hundreds of individual objects and then sticking them in-world together. No matter how many surfaces it has, it is still just one object for the processor to initialize and then worry about, instead of 50. Another issue I've noticed with current tech's polysoup is that the colour/tone/intensity of the textures aren't as vibrant as they are when applied to DIF. I expect this is down to the lightmaps.
Anyhow, it'll be interesting to see what the new tech brings. Roll on Beta.
About the author
One Bloke ... In His Bedroom ... Making Indie Games ...
#2
Yeah, I saw your blog (and I'm sure there's a resource somewhere .. maybe TGE152). Of course my DTS/DIF levels are the whole terrain size (or near enough).
Having said that, it's mostly road or rail networks which cause the huge size, linking one village to another. With those (hopefully) taken care of with the new road tools I'd be able to make villages/town-areas into seperate models easier (if I do end up using DTS instead of DIF). edit: Of course DIFs don't vanish due to fogging.
Either that or a quick engine mod declaring a new model class and forcing it to always render. Anyhow, we'll see how the "advanced lighting" affects performance.
I've nothing against DIF, especially now I've realised how to avoid it blowing up on export.
04/06/2009 (3:47 pm)
@AndyYeah, I saw your blog (and I'm sure there's a resource somewhere .. maybe TGE152). Of course my DTS/DIF levels are the whole terrain size (or near enough).
Having said that, it's mostly road or rail networks which cause the huge size, linking one village to another. With those (hopefully) taken care of with the new road tools I'd be able to make villages/town-areas into seperate models easier (if I do end up using DTS instead of DIF). edit: Of course DIFs don't vanish due to fogging.
Either that or a quick engine mod declaring a new model class and forcing it to always render. Anyhow, we'll see how the "advanced lighting" affects performance.
I've nothing against DIF, especially now I've realised how to avoid it blowing up on export.
#3
The main hassle im finding with the Dts's like you say is the texturing/lighting, they either seem too dark, wierdly shadowed, or really washed out and bland from overlighting... its trying to find that compromise point. shadowing etc isnt that noticable on the smaller stuff that dts was previously used for, but on larger stuff is where, to me, it gets alot more noticable, plus as theres no baked lighting passo n a dts, the suns angle etc do not effect structural shadowing on a dts as it does on a dif, and omgosh stuff like stairs on a dts are doing my head in... if the scale isnt right you get stuck all the time. I tried a workaround that eb suggested, making the stairs non colission and having an invisible ramp as the colission just above them, but i feel it looks worse in use because it makes the players model very visibly float above the stairs, or cut into them...
Edit:/ couple pics to show what i mean on the lighting issues im having problems with.
left is dts, right is a portalled dif with no ambient on in the interior, i cant recreate the same kind of lighting and feel within a dts... and like you say it feels like the dts loses the vibrance of the textures...
With controlled entity lighting in a dif even the normal maps and specular to me feels to work hell of alot better on larger stuff like buildings then it does as a dts
thats just a flat dif floor, again in a portalled interior with just entity lighting. I cant recreate something like that on a dts :/ but i have to start using dts's soon as it seems difs are being made obsolete
*hides from jon before he hits me for still moaning about the dif/dts transition*
another edit:/ You from hull aswell steve ? just noticed the black and amber pub :P
04/06/2009 (4:34 pm)
This Dif/Dts comparison is something ive been looking into alot lately aswell. Ive used bsp for mapping for years, but mesh stuff is still new to me, although i feel im learning it pretty fast on my own when ive the time.The main hassle im finding with the Dts's like you say is the texturing/lighting, they either seem too dark, wierdly shadowed, or really washed out and bland from overlighting... its trying to find that compromise point. shadowing etc isnt that noticable on the smaller stuff that dts was previously used for, but on larger stuff is where, to me, it gets alot more noticable, plus as theres no baked lighting passo n a dts, the suns angle etc do not effect structural shadowing on a dts as it does on a dif, and omgosh stuff like stairs on a dts are doing my head in... if the scale isnt right you get stuck all the time. I tried a workaround that eb suggested, making the stairs non colission and having an invisible ramp as the colission just above them, but i feel it looks worse in use because it makes the players model very visibly float above the stairs, or cut into them...
Edit:/ couple pics to show what i mean on the lighting issues im having problems with.
left is dts, right is a portalled dif with no ambient on in the interior, i cant recreate the same kind of lighting and feel within a dts... and like you say it feels like the dts loses the vibrance of the textures...With controlled entity lighting in a dif even the normal maps and specular to me feels to work hell of alot better on larger stuff like buildings then it does as a dts
thats just a flat dif floor, again in a portalled interior with just entity lighting. I cant recreate something like that on a dts :/ but i have to start using dts's soon as it seems difs are being made obsolete
*hides from jon before he hits me for still moaning about the dif/dts transition*
another edit:/ You from hull aswell steve ? just noticed the black and amber pub :P
#4
There's a couple of TGE resources for fading but they only used a rough distance to player calculation for fading so sniper rifles didn't work too well once you scoped in.
I think the road tools are one of the features I'm most looking forward to... trying to line up smaller road sections, T-junctions etc is a real pain right now. Good things to come with Torque 3D
04/06/2009 (4:47 pm)
@Steve - blimey you're building the whole town as one DTS? missed that bit in the first read through.There's a couple of TGE resources for fading but they only used a rough distance to player calculation for fading so sniper rifles didn't work too well once you scoped in.
I think the road tools are one of the features I'm most looking forward to... trying to line up smaller road sections, T-junctions etc is a real pain right now. Good things to come with Torque 3D
#5
04/06/2009 (6:01 pm)
@Andy - they're not getting rid of difs, they did say they were going to put some optimization to make them perform better in T3D though.
#6
Ive heard that too, but also dts's have apparently been majorly improved on the performance, i guess its something we just have to 'wait and see' to make comparisons for ourselves...
04/06/2009 (6:19 pm)
@ Jaimi:Ive heard that too, but also dts's have apparently been majorly improved on the performance, i guess its something we just have to 'wait and see' to make comparisons for ourselves...
#7
And as Jaimi says, it looks like DIF might be getting a new lease of life.
And yep, for my sins I'm from 'ull. Nice wood floor texture!
@Andy Rollins - yes the town is all one huge model. My previous town-model is in DIF only and had about 9700 surfaces (eventually I got NULL to work properly and reduced that to 7000 or so). It works fine, about 85-100 fps (with everything in the level, 500 objects (mainly trees in shapeReplicators), my AI shooting at me, etc).
04/06/2009 (6:34 pm)
@Andy Wright - I've always found that stairs need plenty of room overhead, as the player gets knocked higher than the step height to mount it, and the collision comes from the top of the bounding box and not actual model's head. For this reason when entering a building I now put a step outside, so the player is at the correct level of the floor before getting into the doorframe (and possibly rebounding).And as Jaimi says, it looks like DIF might be getting a new lease of life.
And yep, for my sins I'm from 'ull. Nice wood floor texture!
@Andy Rollins - yes the town is all one huge model. My previous town-model is in DIF only and had about 9700 surfaces (eventually I got NULL to work properly and reduced that to 7000 or so). It works fine, about 85-100 fps (with everything in the level, 500 objects (mainly trees in shapeReplicators), my AI shooting at me, etc).
#8
im just off prinny ave :P lived up here for 2 years now, originally from 'down sarf' tho... paradise... uh nm, luton...
Im gonna sit back and wait and see what the difs are like within t3d tbh, having a bit of a sabatical from ingame stuff, teaching myself mesh based modelling and doing some 2d work, got a new camera/tripod so want to get a new texture library going aswell
prettymuch all your buildings in the town are just exterior shells right ? with a facecount that low for something that large.
04/06/2009 (6:49 pm)
its on dts's that ive had problems with for stairs, when using polysoup the model gets stuck to them, it only seems to be on stairs im getting the problems tho, the ramp trick works but i dont like the resulting look :/ had no problems with dif stairs.im just off prinny ave :P lived up here for 2 years now, originally from 'down sarf' tho... paradise... uh nm, luton...
Im gonna sit back and wait and see what the difs are like within t3d tbh, having a bit of a sabatical from ingame stuff, teaching myself mesh based modelling and doing some 2d work, got a new camera/tripod so want to get a new texture library going aswell
prettymuch all your buildings in the town are just exterior shells right ? with a facecount that low for something that large.
#9
As for your stairs problems, make sure your normals are pointing out and your mesh is not double sided - both errors I've walked into before (and sometimes still do when I forget!).
Luton ain't much more glamorous than Hull...
04/06/2009 (7:01 pm)
Most of my buildings are shells, a couple aren't, the textures are facades giving the illusion of 2 floors, windows and doors.As for your stairs problems, make sure your normals are pointing out and your mesh is not double sided - both errors I've walked into before (and sometimes still do when I forget!).
Quote:got a new camera/tripod so want to get a new texture library goingThat's what I bought a new camera for too! And of course there's always www.cgtextures.com for raiding every-so-often.
Luton ain't much more glamorous than Hull...
#10
04/06/2009 (7:12 pm)
heh Hulls like heaven compared to luton... 
#11
Hey Andy... let me put it in terms you can understand... "Difs are dodgey, mate, I can't be bovered. Yeah Baby, shagadelic!"
Heheh, you know I am jk and bff's and stuff.
But I think the OP makes the point I was trying to make but he used pictures and numbers, which can have a chilling effect.
04/06/2009 (8:23 pm)
Quote:*hides from jon before he hits me for still moaning about the dif/dts transition*
Hey Andy... let me put it in terms you can understand... "Difs are dodgey, mate, I can't be bovered. Yeah Baby, shagadelic!"
Heheh, you know I am jk and bff's and stuff.
But I think the OP makes the point I was trying to make but he used pictures and numbers, which can have a chilling effect.
#12
the difference between 160 and 120 fps is what exactly ? anything over 30 is playable, 60 is very smooth... picking at something like that with that kind of fps performance is a bit stingy if you ask me when at the 'lower' fps of only 120 will have better interior lighting ? or do dts's now recieve proper shadowing on themselves from the sun ?
04/06/2009 (8:33 pm)
:Pthe difference between 160 and 120 fps is what exactly ? anything over 30 is playable, 60 is very smooth... picking at something like that with that kind of fps performance is a bit stingy if you ask me when at the 'lower' fps of only 120 will have better interior lighting ? or do dts's now recieve proper shadowing on themselves from the sun ?
#13
Yeah go watch the video again where I ask hard-hitting questions. You can clearly see the trees and such are very fancily lit, to use the technical term.
I am no mathematician, but 120 fps down from 160 is like a 50% loss... MAY AS WELL BE! If you are gonna overlook that much of a loss, just think of when you add, hmmm...a game? to your game... it adds up!
04/06/2009 (8:53 pm)
Quote:or do dts's now recieve proper shadowing on themselves from the sun ?
Yeah go watch the video again where I ask hard-hitting questions. You can clearly see the trees and such are very fancily lit, to use the technical term.
I am no mathematician, but 120 fps down from 160 is like a 50% loss... MAY AS WELL BE! If you are gonna overlook that much of a loss, just think of when you add, hmmm...a game? to your game... it adds up!
#14
I'm no artist, and I'm colorblind to boot, but I've always thought there was something off about the appearance of the same object done in both .dif & .dts format. Glad to see I'm not as delusional as I thought I was!
@Jondo - just to clarify some math: a 50% loss from 160 would result in 80. But you do make a good point about how that little bit extra can add up once you start adding the game to your game.
04/06/2009 (9:48 pm)
Interesting numbers there Steve. I've been liking your long view distance and large area levels for a while now. Have you thought about using a Megaterrain and linking up two or more cities? Just to see, of course. I'm no artist, and I'm colorblind to boot, but I've always thought there was something off about the appearance of the same object done in both .dif & .dts format. Glad to see I'm not as delusional as I thought I was!
@Jondo - just to clarify some math: a 50% loss from 160 would result in 80. But you do make a good point about how that little bit extra can add up once you start adding the game to your game.
#15
Thanks. I have since consulted with someone and they said that would be a 25% loss.
Yeeeouch! if you are using a lower-end card.
04/06/2009 (11:08 pm)
@MichaelThanks. I have since consulted with someone and they said that would be a 25% loss.
Yeeeouch! if you are using a lower-end card.
#16
04/07/2009 (4:24 am)
Here's a question that isn't directly related to the blog, but is related to what is contained here. Does anyone know of a good Map to AnyotherShapeFormat converter? Ultimate Unwrap 3D does have a map loader, but it doesn't work on Vista, which I'm usig on both boxes here. Would be nice to be able to convert some of my old maps to Collada, Obj, 3ds, etc and see how they perform. The only program I've found that works though is the old Map23DS but that doesn't do texturing, which is a major negative.
#17
You'll most likely have to bugger about with the UV's and retexture them though
Havent had any problems with the geometry tho
btw, If your map has a tiled texture, it isnt going to work like that on a dts, it will stretch the single texture over the whole face
04/07/2009 (4:53 am)
Ive been using a semi pain in the ass route of import the map into the source sdk (shhhh) export to vmf, import into xsi and export to either collada or dts.You'll most likely have to bugger about with the UV's and retexture them though
Havent had any problems with the geometry tho
btw, If your map has a tiled texture, it isnt going to work like that on a dts, it will stretch the single texture over the whole face
#18
Just thought, thought you said you have null's working now, are you only using them top of the roof's etc ? or are you also doing in insides of the house shells etc, also are you using any detail brushes?
04/07/2009 (5:08 am)
Steve;Just thought, thought you said you have null's working now, are you only using them top of the roof's etc ? or are you also doing in insides of the house shells etc, also are you using any detail brushes?
#19
I've managed to get from map to shape clean and retain texture co-ords, etc. I go: import map into Cartography Shop (or start with source file in CS). Then I export as .X file. If I import that .X file into MS3D it messes up the UVs, but if I open it with UU3D, it works fine. Save out from there.
Basically same as Andy, but with different apps.
04/07/2009 (5:38 am)
@J.C.I've managed to get from map to shape clean and retain texture co-ords, etc. I go: import map into Cartography Shop (or start with source file in CS). Then I export as .X file. If I import that .X file into MS3D it messes up the UVs, but if I open it with UU3D, it works fine. Save out from there.
Basically same as Andy, but with different apps.
#20
@Michael Hall - no, you're not dillussional, there is a massive difference in how the textures look on a DTS to a DIF. I haven't tried megaterrains with these models but I did have a look at them when TGEA1.7 first came out and found the performance was poor just from the terrain having to render 4 times. Enlarging the size of a single terrain didn't have this drawback, but of course it means less detail in the surface, but at least the texture can be scaled so that doesn't look so bad.
@Jondo - "Yeeeouch! if you are using a lower-end card. " - my performance paranoia is derived from lower end specs. I'm on a 7900GS card, not exactly high spec. I keep my monitor on a lower resolution (1280x768 instead of 1680x1050) because I can't read text on the internet at full res.
On full-res DTS is 106 and DIF is 71 (averages conducted as in blog). So yeah, still around 25-30%.
(I have tried the DIF on a lowspec laptop, gets 20fps, terrain textures are a psychedlic mess - but it does technical play. 1.3dualcore, 385intel integral card)
@Andy Wright - NULL is applied to anything the player won't see. The roads, paths and interior floors are the only structural brushes, everything else is detail. No portalling, I found it causes a judder every time the engine has to make a check, especially with the level being 95% outside, hence so few structural brushes, just what he player stands on.
Export was done with the hacked map2dif_plus_tgea by Alan James, there's another version by Andy Rollins, which Alan says isn't a hack.
04/07/2009 (7:43 am)
Yawn ... morning all (okay, mid-afternoon)@Michael Hall - no, you're not dillussional, there is a massive difference in how the textures look on a DTS to a DIF. I haven't tried megaterrains with these models but I did have a look at them when TGEA1.7 first came out and found the performance was poor just from the terrain having to render 4 times. Enlarging the size of a single terrain didn't have this drawback, but of course it means less detail in the surface, but at least the texture can be scaled so that doesn't look so bad.
@Jondo - "Yeeeouch! if you are using a lower-end card. " - my performance paranoia is derived from lower end specs. I'm on a 7900GS card, not exactly high spec. I keep my monitor on a lower resolution (1280x768 instead of 1680x1050) because I can't read text on the internet at full res.
On full-res DTS is 106 and DIF is 71 (averages conducted as in blog). So yeah, still around 25-30%.
(I have tried the DIF on a lowspec laptop, gets 20fps, terrain textures are a psychedlic mess - but it does technical play. 1.3dualcore, 385intel integral card)
@Andy Wright - NULL is applied to anything the player won't see. The roads, paths and interior floors are the only structural brushes, everything else is detail. No portalling, I found it causes a judder every time the engine has to make a check, especially with the level being 95% outside, hence so few structural brushes, just what he player stands on.
Export was done with the hacked map2dif_plus_tgea by Alan James, there's another version by Andy Rollins, which Alan says isn't a hack.
Torque Owner Andy Rollins
ZDay Game
One solution we've implemented to the vanishing models when the fog hits is to fade them out before they hit that, I posted a little note on it in my last blog, it's given us quite the performance boost as a lot of the smaller items (barrels, crates, etc) don't need to be drawn nearly so often.