DotNetTorque Temporarily Delayed
by Vince Gee · 07/10/2012 (4:28 am) · 21 comments
Good Morning,
After doing many load tests we have discovered an issue within DotNetTorque which in my opinion is a show stopper for the planned release. While DotNetTorque runs great 99 percent of the time, there is one situation which brings the T3D client to a halt. This issue does not affect dedicated servers. It only appears to be an issue when there is a great deal of rendering and sound going on and the CPU utilization is maxed.
It is at this point, the whole client becomes unresponsive. The CSharp is still running fine, but the torque_enginetick() function hangs and never returns. It seems to be hanging in the processevents function inside the engine tick function, but since DotNet is wrapping the T3D dll, debugging c++ is difficult at best.
From my research, there appears to be a known issue when a Dot Net CLR application wraps an unmanaged DLL, and the unmanaged DLL spikes the CPU (or ram). From my reading the two do not play well together at that point and the CLR application to unmanaged DLL mapping will become unstable.
The only way up till now I had to prevent this from happening was to delay the "$platform::timeManagerProcessInterval". On my computer, increasing this interval to 28ms reduced the CPU utilization on average down to about 50 percent. The zone I was loading into had 100 AI running around shooting rifles and grenade launchers in a somewhat tight area. When about 50-75 of the AI where in ghosting range, my CPU would go up to about 90 percent, and then recover back at 50 percent.
Unfortunately, this change also soft capped my FPS to about 37. So, yes it stabilized the client, and it was still playable, considering that original TV's refreshed at 32 FPS. I just feel that this solution isn't acceptable.
Of course, there is more. I ran the TorqueScript client against the same dedicated server and it was even less responsive then the slowed down DotNetTorque client. So on one hand I felt that I had a better client even with slowing down the interval, but on the other hand, artificially slowing down the engine seemed, well like a hack.
Having all of this research and data available to me now I believe I have found a solution which will solve the issue at hand. This solution will allow the T3D.exe and DotNetTorque each to run in their own application space. Thus, the spiking of the T3D.exe application will no longer affect the CLR runtime instance running DotNetTorque.
I have already started converting the linkage over to the new method and plan to have a new release candidate out in a couple weeks. The new release will instead use "Hooking" to couple the two applications together.
The only potential downside to the new solution is that anyone who wishes to release a game professionally will need to digitally sigh there code with a certificate from a public certificate publisher (i.e. Verisign, etc.) This will prevent Windows from considering the code unsafe and throwing security issues.
I apologize for the delay in the release, but believe me when I say I spend every waking moment debugging, testing, and writing DotNetTorque. And I refuse to release a product which I feel is not stable 100 percent of the time.
Vince
After doing many load tests we have discovered an issue within DotNetTorque which in my opinion is a show stopper for the planned release. While DotNetTorque runs great 99 percent of the time, there is one situation which brings the T3D client to a halt. This issue does not affect dedicated servers. It only appears to be an issue when there is a great deal of rendering and sound going on and the CPU utilization is maxed.
It is at this point, the whole client becomes unresponsive. The CSharp is still running fine, but the torque_enginetick() function hangs and never returns. It seems to be hanging in the processevents function inside the engine tick function, but since DotNet is wrapping the T3D dll, debugging c++ is difficult at best.
From my research, there appears to be a known issue when a Dot Net CLR application wraps an unmanaged DLL, and the unmanaged DLL spikes the CPU (or ram). From my reading the two do not play well together at that point and the CLR application to unmanaged DLL mapping will become unstable.
The only way up till now I had to prevent this from happening was to delay the "$platform::timeManagerProcessInterval". On my computer, increasing this interval to 28ms reduced the CPU utilization on average down to about 50 percent. The zone I was loading into had 100 AI running around shooting rifles and grenade launchers in a somewhat tight area. When about 50-75 of the AI where in ghosting range, my CPU would go up to about 90 percent, and then recover back at 50 percent.
Unfortunately, this change also soft capped my FPS to about 37. So, yes it stabilized the client, and it was still playable, considering that original TV's refreshed at 32 FPS. I just feel that this solution isn't acceptable.
Of course, there is more. I ran the TorqueScript client against the same dedicated server and it was even less responsive then the slowed down DotNetTorque client. So on one hand I felt that I had a better client even with slowing down the interval, but on the other hand, artificially slowing down the engine seemed, well like a hack.
Having all of this research and data available to me now I believe I have found a solution which will solve the issue at hand. This solution will allow the T3D.exe and DotNetTorque each to run in their own application space. Thus, the spiking of the T3D.exe application will no longer affect the CLR runtime instance running DotNetTorque.
I have already started converting the linkage over to the new method and plan to have a new release candidate out in a couple weeks. The new release will instead use "Hooking" to couple the two applications together.
The only potential downside to the new solution is that anyone who wishes to release a game professionally will need to digitally sigh there code with a certificate from a public certificate publisher (i.e. Verisign, etc.) This will prevent Windows from considering the code unsafe and throwing security issues.
I apologize for the delay in the release, but believe me when I say I spend every waking moment debugging, testing, and writing DotNetTorque. And I refuse to release a product which I feel is not stable 100 percent of the time.
Vince
About the author
www.winterleafentertainment.com
Torque Owner John Bergman
Default Studio Name