Severe performance issues - DIF vs. DTS - shadows?
by Konrad Kiss · in Torque Game Engine Advanced · 06/26/2008 (7:01 am) · 17 replies
Hi all!
I have found several natural cases where my fps dropped to like 10% or less than what it was originally. Here are the cases:
I have created a dts flying object and a megaterrain with an ordinary dif ground object on it. I realized, that when I flew the dts vehicle close to the dif object, my framerate dropped significantly.
When I rescaled the dts flying vehicle from [1 1 1] to [3 3 3], this performance hit was even more evident (dropped to somewhere 10% and 1% of the original fps value) when flying close to the dif object.
I also felt a smaller perf hit when flying close to the terrain. This again was stronger when the vehicle was scaled to appear larger.
After this, I rescaled my flying vehicle to be [0.3 0.3 0.3]. I noticed that the framerate was the best of all the cases so far.
Following this, I used a number of polysouped dts shapes (approx the same tri count as the dif shape) and then flew close to these. This had no impact on performance at all.
Why does scaling have such a grave effect on performance? All I can think is shadows, but still, that can not be a reason for such a bog-down.
So all this led me to believe that difs, dts objects and shadows don't mix well. Does anybody know what the reason to this performance hit could be? What would a possible solution be? I'd have no problem converting the difs to dts, though I only have the map files for these, and I have no idea how to create dts files from map files.
Any help would be very much appreciated!
I have found several natural cases where my fps dropped to like 10% or less than what it was originally. Here are the cases:
I have created a dts flying object and a megaterrain with an ordinary dif ground object on it. I realized, that when I flew the dts vehicle close to the dif object, my framerate dropped significantly.
When I rescaled the dts flying vehicle from [1 1 1] to [3 3 3], this performance hit was even more evident (dropped to somewhere 10% and 1% of the original fps value) when flying close to the dif object.
I also felt a smaller perf hit when flying close to the terrain. This again was stronger when the vehicle was scaled to appear larger.
After this, I rescaled my flying vehicle to be [0.3 0.3 0.3]. I noticed that the framerate was the best of all the cases so far.
Following this, I used a number of polysouped dts shapes (approx the same tri count as the dif shape) and then flew close to these. This had no impact on performance at all.
Why does scaling have such a grave effect on performance? All I can think is shadows, but still, that can not be a reason for such a bog-down.
So all this led me to believe that difs, dts objects and shadows don't mix well. Does anybody know what the reason to this performance hit could be? What would a possible solution be? I'd have no problem converting the difs to dts, though I only have the map files for these, and I have no idea how to create dts files from map files.
Any help would be very much appreciated!
About the author
http://about.me/konrad.kiss
#2
Thanks for your response. The scene is pretty empty, so it's not the GPU issue, I'm sure. The collision problem could be it, I'm not sure. I have tried it with the default orc player character as the player dts object, and a dif object from Tim Aste's content pack.
Here are some screenshots from the scene with fps data and player scale.. the larger the object scale the higher the performance hit.
I think there's something sinister going on..
06/26/2008 (8:58 am)
Hi Stefan,Thanks for your response. The scene is pretty empty, so it's not the GPU issue, I'm sure. The collision problem could be it, I'm not sure. I have tried it with the default orc player character as the player dts object, and a dif object from Tim Aste's content pack.
Here are some screenshots from the scene with fps data and player scale.. the larger the object scale the higher the performance hit.
I think there's something sinister going on..
#3
06/26/2008 (10:15 am)
That sure looks fishy. Not sure what's going on, can you try to profile it?
#4
Maybe a stupid question, forgive me: What do you mean by / How do I profile it?
06/26/2008 (10:20 am)
I need to add, that the 4.7 FPS is not constant, it's after I move my player character, and - possibly - shadows must be recomputed. Maybe a stupid question, forgive me: What do you mean by / How do I profile it?
#5
Have you tried making the DIF there into a DTS? Also, this happens when you are *next* to it, but not actually colliding? The flying vehicle issue could be related to the atrocious collision response algorithm (a *very* hacky fix for the collisions slowing things down is to simply remove the outer do {} while() loop in the resolveCollisions function of vehicle.cpp... It really doesn't make much of a difference, because by the second inner for loop through the contacts, the resulting impulses are too small to matter, really).
If you're serious about making a vehicle-related game, I'd suggest implementing one of the physics libraries and attaching an object to your vehicle, instead of using the current code for collision. If your game is going to be single player, the current PhysX resource should work fine. Otherwise you'd need to do some fixes to it so it creates a physics sim for the client and the server separately.
06/26/2008 (11:23 am)
It's either CTRL-F3 or ALT-F3 and will output a file into your game directory containing a profile (a text file that has the time a given function or block of code took up in rendering the frame) showing what was taking up the most time in the simulation. Just press and hold for a few seconds, and you should see the .txt file appear. You should open it in WordPad instead of Notepad, since the latter doesn't format it correctly.Have you tried making the DIF there into a DTS? Also, this happens when you are *next* to it, but not actually colliding? The flying vehicle issue could be related to the atrocious collision response algorithm (a *very* hacky fix for the collisions slowing things down is to simply remove the outer do {} while() loop in the resolveCollisions function of vehicle.cpp... It really doesn't make much of a difference, because by the second inner for loop through the contacts, the resulting impulses are too small to matter, really).
If you're serious about making a vehicle-related game, I'd suggest implementing one of the physics libraries and attaching an object to your vehicle, instead of using the current code for collision. If your game is going to be single player, the current PhysX resource should work fine. Otherwise you'd need to do some fixes to it so it creates a physics sim for the client and the server separately.
#6
Not sure how much of the shadow calculations are done on the CPU in TGEA. I think they moved alot of that into a Pixel Shader.
Link to Profiler documentation.
06/26/2008 (11:23 am)
There's a tool integrated into Torque which is called the Profiler. You can enable it and it will watch certain preset (by the programmer) instructions and see where Torque is spending most of it's time. This will obviously not help much if you're GPU bound since the Profiler only can do so much, but at least you can rule out different areas this way.Not sure how much of the shadow calculations are done on the CPU in TGEA. I think they moved alot of that into a Pixel Shader.
Link to Profiler documentation.
#7
06/26/2008 (11:36 am)
The shadows...aren't very optimal, to say the least ;) They have a pretty long path (lots of function overhead) to actually rendering, and it ends up rendering the mesh again.
#8
Wow, the profiler is an excellent tool! :) Love at first sight.
The problem happens next to it as well, but it all makes sense now. Btw, I am using vehicles in the game, but I don't need true physics at all, they are just part of the story.. means of transportation and warfare, but really simple physics will do fine. Anyway, Profiler info:
Here's a list of a normal scene's most demanding functions:
Here's the problem scene's dump:
It's pretty evident now... I think I'm going to spend my time optimizing that part.. or at least will try to. I have not yet converted the DIF to DTS, but first I'll try to do it the hard way.. afterall this is not how it's supposed to work.
I have always wanted to know more about shaders.. I guess it's just the right time to start experimenting.
The weird thing is that object scaling seems to be affecting the whole issue too much imo.
06/26/2008 (12:21 pm)
@Ross & Stefan:Wow, the profiler is an excellent tool! :) Love at first sight.
The problem happens next to it as well, but it all makes sense now. Btw, I am using vehicles in the game, but I don't need true physics at all, they are just part of the story.. means of transportation and warfare, but really simple physics will do fine. Anyway, Profiler info:
Here's a list of a normal scene's most demanding functions:
27.857 29.599 81249 TerrainRender_renderChunkNormal 12.731 18.085 72 TerrainRenderRecurse 4.924 5.264 72 SkyRender 4.395 4.395 41302 ClipMap_calculateClipMapLevels
Here's the problem scene's dump:
83.419 83.418 7 sgShadowProjector_sgCalculateBoundingBox_Part 5.351 9.702 170 AdvanceObjects 2.727 2.727 14043 ContainerCastRay 1.095 1.095 7 sgShadowProjector_sgCalculateBoundingBox_find
It's pretty evident now... I think I'm going to spend my time optimizing that part.. or at least will try to. I have not yet converted the DIF to DTS, but first I'll try to do it the hard way.. afterall this is not how it's supposed to work.
I have always wanted to know more about shaders.. I guess it's just the right time to start experimenting.
The weird thing is that object scaling seems to be affecting the whole issue too much imo.
#9
06/26/2008 (12:39 pm)
Before you dive into that code (which is fairly heinous IMO (I'm looking at you Kabus! :P)) I'd see if changing it to a DTS has any effect. I have a feeling it probably goes through a different path if that's the case.
#10
So right now, I'm even having a problem setting blender where to export. I tried using quick export, but I could not find the file exported anywhere - it never asked for an output filename.
As soon as I have the dts ready, I'll paste another Profiler log here.
Do you guys know of a simple way to convert from MAP or DIF to DTS? I feel sorry for artists now, really.
06/26/2008 (2:36 pm)
I've so far failed at converting the dif or map file to dts. Im trying Deep Exploration to convert map to 3ds / dxf or something more common, and then blender to convert that to dts - although I'm the ideal noob to blender, and the interface is really alien to me - nothing like Windows or OS X. *sigh*So right now, I'm even having a problem setting blender where to export. I tried using quick export, but I could not find the file exported anywhere - it never asked for an output filename.
As soon as I have the dts ready, I'll paste another Profiler log here.
Do you guys know of a simple way to convert from MAP or DIF to DTS? I feel sorry for artists now, really.
#11
06/26/2008 (3:02 pm)
@Konrad, I'm not an artist either, so I really have no idea. Making/working with DIFs is arcane magic to me. 
#12
You can covert the .MAP format with DeleD, the Torque .MAP plugin for DeleD and Milkshape. I'm not sure if you have either of these applications, but DeleD Lite is free, and there is still (I believe) a working 30 day demo of Milkshape. It would still require you to purchse the $14.95 map importer that is for DeleD, but it works fine with the Lite version.
Basically what you have to do is import the .MAP into DeleD, then export it as a .X file (that seems to be the easiest format to work with when importing it into Milkshape...I've tried .OBJ and .3DS, sometimes they misbehave). Start up Milkshape and use the DX mesh tools from the Tools drop down menu, if the .X is not to outrageously complicated (since you started with a .MAP/.DIF, it's probably not, but I've occasionally had ones lock up Milkshape...really complicated ones).
Once it loads you use DTSPlus! exporter (I believe the latest is the one by Chris Robertson on the ChumbaLum Soft website). Your DTS object will not have a collision mesh, but I guess you can use the polysoup check box setting - I haven't messed around too much lately with TGEA, or you can build collison boxes, i.e. box meshes that roughly match the models surface with the name "collision-1, collision-2, etc.
I know that perhaps sounds complicated, but it works about 95% of the time.
06/26/2008 (5:04 pm)
@Konrad,You can covert the .MAP format with DeleD, the Torque .MAP plugin for DeleD and Milkshape. I'm not sure if you have either of these applications, but DeleD Lite is free, and there is still (I believe) a working 30 day demo of Milkshape. It would still require you to purchse the $14.95 map importer that is for DeleD, but it works fine with the Lite version.
Basically what you have to do is import the .MAP into DeleD, then export it as a .X file (that seems to be the easiest format to work with when importing it into Milkshape...I've tried .OBJ and .3DS, sometimes they misbehave). Start up Milkshape and use the DX mesh tools from the Tools drop down menu, if the .X is not to outrageously complicated (since you started with a .MAP/.DIF, it's probably not, but I've occasionally had ones lock up Milkshape...really complicated ones).
Once it loads you use DTSPlus! exporter (I believe the latest is the one by Chris Robertson on the ChumbaLum Soft website). Your DTS object will not have a collision mesh, but I guess you can use the polysoup check box setting - I haven't messed around too much lately with TGEA, or you can build collison boxes, i.e. box meshes that roughly match the models surface with the name "collision-1, collision-2, etc.
I know that perhaps sounds complicated, but it works about 95% of the time.
#13
Thank you. Too bad there's no free alternative for converting between Torque's two main 3d object formats.
I like DeleD a lot, so I might go for the Torque pack afterall.
06/27/2008 (1:09 am)
@Alan,Thank you. Too bad there's no free alternative for converting between Torque's two main 3d object formats.
I like DeleD a lot, so I might go for the Torque pack afterall.
#15
Console:
I'm worried about having to debug the entire Internet next. Ahh... *sigh*... life's hard.
06/27/2008 (6:18 am)
@Joseph: Thanks for the idea! I managed to export the .lwo with constructor, but the import failed in Blender:Console:
This is: lwo_import 0.4.0 Importing file: C:\...\sf6k.lwo Traceback (most recent call last): File "<string>", line 169, in read File "<string>", line 329, in read_lwo2 File "<string>", line 902, in read_surfs File "C:\Program Files\Blender Foundation\Blender\python25.zip\struct.py", lin e 87, in unpack struct.error: unpack requires a string argument of length 4
I'm worried about having to debug the entire Internet next. Ahh... *sigh*... life's hard.
#17
06/27/2008 (7:31 am)
:) I tried to export DXF from Hammer - it should know .map files pretty well.. but then again, the Blender DXF import failed (both importers failed). Whou.. happy daaay. 
Torque Owner Stefan Lundmark
Sounds like you're either having performance issues because of collisions, or because the GPU is getting too much thrown at it. The former sounds most likely, to me, since when you scale your vehicle up you're getting closer to the collision hull of the interior/polysoup and the polylists will start to do their work earlier than normally.
However, one object colliding with another shouldn't ever give such a huge performance hit. Is your collision hull Convex? Simply throw in a player (which has very basic collision detection) and run up to the building and see if you get low framerate. If not, you can rule out rendering - which is good.
Failing that, do also make sure you're portalizing your interiors if they're big.