DIFs slow in 1.8.1 (and all previous version of TGEA)
by Greg G · in Torque Game Engine Advanced · 02/13/2009 (12:45 pm) · 17 replies
I haven't noticed anyone talking specifically about this on the forums so I decided to post here.
For my test I made a building in both Maya and Constructor both models contained about 1300 verts, and both used about the same textures.
I made 60 copies of both the DTS and DIF models, and separated then into two groups. When I look at the DTS group I get 200 FPS
When I look at the DIF group I get 45 FPS
Aren't DIFs suppose to be faster than DTS? I realize that difs may not be being batch rendered, but I compared the same dif group of buildings in TGE 1.5.2 and TGEA 1.8.1 & and the difference was night and day. I got almost 100 FPS in TGE 1.5.2 (this was on a different machine).
Finally I thought it might because of my shadow quality on the models but I turned that way way down, the shadows are basically blobs and I get the same results.
Does anyone have any idea why this happening?
For my test I made a building in both Maya and Constructor both models contained about 1300 verts, and both used about the same textures.
I made 60 copies of both the DTS and DIF models, and separated then into two groups. When I look at the DTS group I get 200 FPS
When I look at the DIF group I get 45 FPS
Aren't DIFs suppose to be faster than DTS? I realize that difs may not be being batch rendered, but I compared the same dif group of buildings in TGE 1.5.2 and TGEA 1.8.1 & and the difference was night and day. I got almost 100 FPS in TGE 1.5.2 (this was on a different machine).
Finally I thought it might because of my shadow quality on the models but I turned that way way down, the shadows are basically blobs and I get the same results.
Does anyone have any idea why this happening?
#2
02/13/2009 (2:59 pm)
No there was no collision, I enabled polysoup on a couple of them, but other than that nothing. Collision is not important for my project as all units are controlled over the network and collision is handled by 3rd party software on the server. 
#3
02/13/2009 (5:35 pm)
I remember reading this somewhere else, that DIFs while still usualable, were being phased out in favor of polysoup and such solutions. There has a been a few posts regarding Constructor, and loosing it as a tool with DIFs becoming less useful with the new TGEA setups. 
#4
The numbers here are legitimate. One of our first maps had a large DIF city, and whenever you viewed it, even in the lowest LOD version, which was pretty low on polys, the frame rate would plummet. We wound up using Polysoup models for the future maps and the frame rate was much better, despite having no portals. The two areas that DIF still does things well though is the lightmaps and the ability to cull DTS shapes inside of portals. Those features will be missed if DIFs are removed and no replacement is found. Otherwise though I think its probably a good thing to see this format go away, it's definately more difficult to create good looking models with.
02/13/2009 (5:45 pm)
In the large T3D recommendation thread GG alluded to DIF probably being phased out, but maybe having a DIF loader which loaded them as DTS files. The numbers here are legitimate. One of our first maps had a large DIF city, and whenever you viewed it, even in the lowest LOD version, which was pretty low on polys, the frame rate would plummet. We wound up using Polysoup models for the future maps and the frame rate was much better, despite having no portals. The two areas that DIF still does things well though is the lightmaps and the ability to cull DTS shapes inside of portals. Those features will be missed if DIFs are removed and no replacement is found. Otherwise though I think its probably a good thing to see this format go away, it's definately more difficult to create good looking models with.
#5
But regardless, while i dont disagree that DTS models look better... Removing as a competitive option seems a bit backward. I mean, if its required to make DTS the dominint format, then I can understand that.. But either keep it in and support it, or take it out and make us make the hard choice. Making it slowly less effective seems like someone is trying to rob peter to pay paul and make everyone happy without telling everyone the format is really dying.
02/13/2009 (6:12 pm)
Which, imo really cuts alot of the tools out that i have found really comfortable and other have as well. Essential DTS models being used with poly soup being the only solution out of the box for decent numbers. I mean, really what else is going to be cut out? I know a lots of folks that have gotten really good with 3dws to make use of the DIFs, not to mention constructor. Does anyone know if GG is going to provide some kind of converter to make Constuctor or a DTS constructor setup. In GG defense i see that they have created the start of a interesting video tutorial, but will it be made complete and comprehensive? Id be happy to pay for a tutorial set that i can download and/or buy (as a note i am on 56k, so downloading something the standard method is not a option here) But regardless, while i dont disagree that DTS models look better... Removing as a competitive option seems a bit backward. I mean, if its required to make DTS the dominint format, then I can understand that.. But either keep it in and support it, or take it out and make us make the hard choice. Making it slowly less effective seems like someone is trying to rob peter to pay paul and make everyone happy without telling everyone the format is really dying.
#6
02/13/2009 (6:30 pm)
I would be happy to move on from DIF for my project if it were not for how easy shadows are generated with them. Establishing a good art pipeline for getting shadows on models in a scene is very difficult with DTS objects. Is anyone aware of any alternatives or perhaps some way of getting self shadowing on DTS objects?
#7
If all of your DTSes didn't have collision (but of course DIFs do) you can't call it a proper test. Give them all collision (polysoup or meshed) and then see the difference.
As has been mentioned above, it seems DIF will be replaced for T3D/ Torque2/Juggernaut/thingymabob with a model/mesh based solution that features something like lightmaps (another reason I spent so long experimenting with DIF and Constructor).
Frankly if they'd clear up the brush border issue with the tgea DIF exporter I'd be a very happy bunny, but in the meantime I'll live with it.
My tupence and ha'penny bit
02/13/2009 (7:44 pm)
I found that making one big DIF (exported with map2dif_plus_tgea - it comes with TGEA under engine-tools, read through the constructor forum for info on a few issues) gave the best performance. I found I just had too much "stuff" for polysoup to work well (lagged terribly). Building your DIF with the rules of "old-school-BSP" construction also helps. And if your player isn't going to leave the area, there is no point making LODs.If all of your DTSes didn't have collision (but of course DIFs do) you can't call it a proper test. Give them all collision (polysoup or meshed) and then see the difference.
As has been mentioned above, it seems DIF will be replaced for T3D/ Torque2/Juggernaut/thingymabob with a model/mesh based solution that features something like lightmaps (another reason I spent so long experimenting with DIF and Constructor).
Frankly if they'd clear up the brush border issue with the tgea DIF exporter I'd be a very happy bunny, but in the meantime I'll live with it.
My tupence and ha'penny bit
#8
Faster in what way? Are you colliding with those DIF's? No? Then you're comparing apples to oranges. DIF's still have faster collision checks, but you're limited to convex faces.
They are batch rendered per material.
This is a good question! You could profile it and note if your're CPU or GPU bound. DIF's perform other calculations than what a TSShape does, but today these calculations won't save you a whole lot of fillrate. This is probably why they are replacing it.
Properly portalized interiors are still heaps faster than the same geometry in polysoup.
02/14/2009 (2:34 am)
Quote:
Aren't DIFs suppose to be faster than DTS?
Faster in what way? Are you colliding with those DIF's? No? Then you're comparing apples to oranges. DIF's still have faster collision checks, but you're limited to convex faces.
Quote:
I realize that difs may not be being batch rendered,
They are batch rendered per material.
Quote:
Does anyone have any idea why this happening?
This is a good question! You could profile it and note if your're CPU or GPU bound. DIF's perform other calculations than what a TSShape does, but today these calculations won't save you a whole lot of fillrate. This is probably why they are replacing it.
Properly portalized interiors are still heaps faster than the same geometry in polysoup.
#9
Profile and then maybe a reason/cause for your slowdown can be found.
02/14/2009 (4:55 am)
This actually comes up every so often...Quote:For my test I made a building in both Maya and Constructor both models contained about 1300 verts, and both used about the same textures.Due to the differences in brush based and mesh based modeling, no matter how similar you think things are you really can't compare the two side by side.
Quote:Properly portalized interiors are still heaps faster than the same geometry in polysoup.The very words I would have said... perhaps I would have said "should be heaps faster" instead.
Profile and then maybe a reason/cause for your slowdown can be found.
#10
As has already been mentioned, DIFs need to be constructed and set up properly.
02/15/2009 (6:11 am)
I have found in my own experiences that the only thing that bogs the engine down that deals with DIFs are dynamic lights. Aside from that, performance is great. As has already been mentioned, DIFs need to be constructed and set up properly.
#11
We are not using a player at all.
So they tests I did were with the spectator camera from a great distance looking down on all the buildings. In this case, would collision still affect the performance?
Also, at the end of the day I understand the entire rendering process for a BSP tree is different from that of a DTS, that's why DIFs are suppose to be more effective and efficient in certain circumstances. It seams completely reasonable that there would be a difference between the two object types but given their purpose, which is to be rendered in real time and provided acceptable frame rates, I would not expect this kind of gap in performance.
Also, as I mentioned before, the tests I did between TGE and TGEA show that the drop in frame rate with DIFs is MASSIVE between the two engines.
Finally, I spoke to Matt Fairfax about this and he stated that "DIF support in TGEA is abysmal" and that he will be looking into in Torque 3D.
02/15/2009 (1:32 pm)
Perhaps I am mistaken, but I thought that unless the player is close to something with collision then collision is not active; in other words no collisions calculations are being done unless the player is within a certain distance from the collidable object.We are not using a player at all.
So they tests I did were with the spectator camera from a great distance looking down on all the buildings. In this case, would collision still affect the performance?
Also, at the end of the day I understand the entire rendering process for a BSP tree is different from that of a DTS, that's why DIFs are suppose to be more effective and efficient in certain circumstances. It seams completely reasonable that there would be a difference between the two object types but given their purpose, which is to be rendered in real time and provided acceptable frame rates, I would not expect this kind of gap in performance.
Also, as I mentioned before, the tests I did between TGE and TGEA show that the drop in frame rate with DIFs is MASSIVE between the two engines.
Finally, I spoke to Matt Fairfax about this and he stated that "DIF support in TGEA is abysmal" and that he will be looking into in Torque 3D.
#12
I think it is worth pointing out that during TGE's (aka V12 aka the engine used by Tribes 2) writing BSP tree format used in DIFs was for the most part a standard. All the game engines of the time used BSP Tree for the most part. Thus when the engine was originaly made in opengl it was the focus of the engine. Most all the buildings used in Tribes was made in dif format. DTS use was limited and only on a as needed type setup I imagine.
When TGEA was being created and pulled out of TGE and reworked in directx, I imagine they tried to embrace DTS support more-so as it was something that almost everyone of the community at the time wanted to use. Thus I do imagine it got alot more dev time to get it upto speed. I imagine less time was spent in speeding up DIF use and getting it upto speed in directx.
With the right people working with the engine you can get some massive speed increases across the board. Atleast from things I have seen done with the engine you can. Simple dts and dif geometry occlusion culling can net some massive speed increases in most cases. (BTW this is something I would like to see Torque 3D pickup - though I do understand it can be a per project type feature as not everyone would want it)
Things to remember with DIFs is that they, for the most part, are designed to be used as buildings and such with proper portaling setup. You can make a full sized city with both interiors and exteriors (with portaling) and net a much higher fps then if you used pure DTS objects for it all. (I've seen it done and have done it before) When using DIFs make sure you use different LODs and find a proper breaking points for the different LODs. Theres really no need to keep a full detailed skyscrapper loaded in view when a simple box would work instead. Texture size plays a HUGE part in DIF speed. You will be hard pressed to load a full city of skyscrappers loaded with details (windows, airconditors, ect.) and keep a full sizes 2056 texture on them (It won't work, I tried it also and it drops the fps like a rock in the ocean). You can use different textures for different lods and do make use of it. Oh and lightmaps - I recommened small sized lightmaps where possible 256 if at all possible as this also has a large effect on dif useage in a level.
02/15/2009 (6:31 pm)
Well comparing TGE to TGEA is like apples to oranges. You are comparing a engine that was designed in opengl to an almost complete rewrite in directx. I am sure there are loads that could be done to improve dif support in TGEA.I think it is worth pointing out that during TGE's (aka V12 aka the engine used by Tribes 2) writing BSP tree format used in DIFs was for the most part a standard. All the game engines of the time used BSP Tree for the most part. Thus when the engine was originaly made in opengl it was the focus of the engine. Most all the buildings used in Tribes was made in dif format. DTS use was limited and only on a as needed type setup I imagine.
When TGEA was being created and pulled out of TGE and reworked in directx, I imagine they tried to embrace DTS support more-so as it was something that almost everyone of the community at the time wanted to use. Thus I do imagine it got alot more dev time to get it upto speed. I imagine less time was spent in speeding up DIF use and getting it upto speed in directx.
With the right people working with the engine you can get some massive speed increases across the board. Atleast from things I have seen done with the engine you can. Simple dts and dif geometry occlusion culling can net some massive speed increases in most cases. (BTW this is something I would like to see Torque 3D pickup - though I do understand it can be a per project type feature as not everyone would want it)
Things to remember with DIFs is that they, for the most part, are designed to be used as buildings and such with proper portaling setup. You can make a full sized city with both interiors and exteriors (with portaling) and net a much higher fps then if you used pure DTS objects for it all. (I've seen it done and have done it before) When using DIFs make sure you use different LODs and find a proper breaking points for the different LODs. Theres really no need to keep a full detailed skyscrapper loaded in view when a simple box would work instead. Texture size plays a HUGE part in DIF speed. You will be hard pressed to load a full city of skyscrappers loaded with details (windows, airconditors, ect.) and keep a full sizes 2056 texture on them (It won't work, I tried it also and it drops the fps like a rock in the ocean). You can use different textures for different lods and do make use of it. Oh and lightmaps - I recommened small sized lightmaps where possible 256 if at all possible as this also has a large effect on dif useage in a level.
#13
Correct, and that's the whole point :) If you didn't use any collisions then there's nothing to compare. If you don't need collisions then it's a whole different story, but most of us do.
02/16/2009 (2:45 am)
Quote:
So they tests I did were with the spectator camera from a great distance looking down on all the buildings. In this case, would collision still affect the performance?
Correct, and that's the whole point :) If you didn't use any collisions then there's nothing to compare. If you don't need collisions then it's a whole different story, but most of us do.
#14
Summary: DIF structures are not the most suitable for being replicated in dozens, since there is a higher cost per instance now that TGEA relies on vertex buffers and batching. If you really need your lightmaps, try merging your structures into less DIF instances and find ways to generate less lightmap pages.
02/16/2009 (6:51 am)
Did your DIFs have lightmapping turned on? That's a key difference: each DIF instance will create a copy of its lightmap. Also, I'm not sure, but I think lightmaps are stored in 256x256 pages. Depending on the lexel density of your difs, they might be using multiple pages, and this means that, even if they are material batched, there's still some texture swapping going on between them. Also, the DIF visibility checks are optimized for large structures, while the DTS ones are designed for small structures.Summary: DIF structures are not the most suitable for being replicated in dozens, since there is a higher cost per instance now that TGEA relies on vertex buffers and batching. If you really need your lightmaps, try merging your structures into less DIF instances and find ways to generate less lightmap pages.
#15
While I am sure this is good advice its humorous because many of the work around for constructor require you to break up your models in order to get them to export. Very large and complex structures have loads of problem being exported and loaded. So if I follow the rules of constructor for export and the rules of TGEA for performance that basically narrows the usefulness of DIFs down to fairly simple structures, consisting of no more than a few hundred brushes, scattered infrequently thought a map so as not to have many on screen at one time.
I think what I am going to wind up doing is only use DIFs for buildings that the camera needs to go inside so that I can use the portals for performance and just build everything else out of DTS and try to find some way of baking in the shadows.
That pretty much seams the only option with the way torque is currently set up. If I had a team of programmers and wasn't so reliant on stock torque I am sure I could take the time to work around it but I dont so I guess this is the best route for me.
02/16/2009 (10:23 am)
Quote:Did your DIFs have light mapping turned on?They had baked shadows, so yes. I am actually not aware of anyway to turn the baked shadows off without making changes in engine.
Quote:...try merging your structures into less DIF instances
While I am sure this is good advice its humorous because many of the work around for constructor require you to break up your models in order to get them to export. Very large and complex structures have loads of problem being exported and loaded. So if I follow the rules of constructor for export and the rules of TGEA for performance that basically narrows the usefulness of DIFs down to fairly simple structures, consisting of no more than a few hundred brushes, scattered infrequently thought a map so as not to have many on screen at one time.
I think what I am going to wind up doing is only use DIFs for buildings that the camera needs to go inside so that I can use the portals for performance and just build everything else out of DTS and try to find some way of baking in the shadows.
That pretty much seams the only option with the way torque is currently set up. If I had a team of programmers and wasn't so reliant on stock torque I am sure I could take the time to work around it but I dont so I guess this is the best route for me.
#16
02/16/2009 (1:11 pm)
Oh, and I know this has been posted elsewhere, but I did find a solution to the shadows issue with DTS models. I am using this process: www.futurenation.net/dtsMerger/tiki-index.php
#17
02/16/2009 (2:02 pm)
Since the same lightmap will be shared among your DTS models, you will still get better performance than DIF models.
Torque Owner TheGasMan
G.A.S. [+others]