DotNetTorque Temporarily Delayed
by Vince Gee · 07/10/2012 (4:28 am) · 21 comments
Good Morning,
After doing many load tests we have discovered an issue within DotNetTorque which in my opinion is a show stopper for the planned release. While DotNetTorque runs great 99 percent of the time, there is one situation which brings the T3D client to a halt. This issue does not affect dedicated servers. It only appears to be an issue when there is a great deal of rendering and sound going on and the CPU utilization is maxed.
It is at this point, the whole client becomes unresponsive. The CSharp is still running fine, but the torque_enginetick() function hangs and never returns. It seems to be hanging in the processevents function inside the engine tick function, but since DotNet is wrapping the T3D dll, debugging c++ is difficult at best.
From my research, there appears to be a known issue when a Dot Net CLR application wraps an unmanaged DLL, and the unmanaged DLL spikes the CPU (or ram). From my reading the two do not play well together at that point and the CLR application to unmanaged DLL mapping will become unstable.
The only way up till now I had to prevent this from happening was to delay the "$platform::timeManagerProcessInterval". On my computer, increasing this interval to 28ms reduced the CPU utilization on average down to about 50 percent. The zone I was loading into had 100 AI running around shooting rifles and grenade launchers in a somewhat tight area. When about 50-75 of the AI where in ghosting range, my CPU would go up to about 90 percent, and then recover back at 50 percent.
Unfortunately, this change also soft capped my FPS to about 37. So, yes it stabilized the client, and it was still playable, considering that original TV's refreshed at 32 FPS. I just feel that this solution isn't acceptable.
Of course, there is more. I ran the TorqueScript client against the same dedicated server and it was even less responsive then the slowed down DotNetTorque client. So on one hand I felt that I had a better client even with slowing down the interval, but on the other hand, artificially slowing down the engine seemed, well like a hack.
Having all of this research and data available to me now I believe I have found a solution which will solve the issue at hand. This solution will allow the T3D.exe and DotNetTorque each to run in their own application space. Thus, the spiking of the T3D.exe application will no longer affect the CLR runtime instance running DotNetTorque.
I have already started converting the linkage over to the new method and plan to have a new release candidate out in a couple weeks. The new release will instead use "Hooking" to couple the two applications together.
The only potential downside to the new solution is that anyone who wishes to release a game professionally will need to digitally sigh there code with a certificate from a public certificate publisher (i.e. Verisign, etc.) This will prevent Windows from considering the code unsafe and throwing security issues.
I apologize for the delay in the release, but believe me when I say I spend every waking moment debugging, testing, and writing DotNetTorque. And I refuse to release a product which I feel is not stable 100 percent of the time.
Vince
After doing many load tests we have discovered an issue within DotNetTorque which in my opinion is a show stopper for the planned release. While DotNetTorque runs great 99 percent of the time, there is one situation which brings the T3D client to a halt. This issue does not affect dedicated servers. It only appears to be an issue when there is a great deal of rendering and sound going on and the CPU utilization is maxed.
It is at this point, the whole client becomes unresponsive. The CSharp is still running fine, but the torque_enginetick() function hangs and never returns. It seems to be hanging in the processevents function inside the engine tick function, but since DotNet is wrapping the T3D dll, debugging c++ is difficult at best.
From my research, there appears to be a known issue when a Dot Net CLR application wraps an unmanaged DLL, and the unmanaged DLL spikes the CPU (or ram). From my reading the two do not play well together at that point and the CLR application to unmanaged DLL mapping will become unstable.
The only way up till now I had to prevent this from happening was to delay the "$platform::timeManagerProcessInterval". On my computer, increasing this interval to 28ms reduced the CPU utilization on average down to about 50 percent. The zone I was loading into had 100 AI running around shooting rifles and grenade launchers in a somewhat tight area. When about 50-75 of the AI where in ghosting range, my CPU would go up to about 90 percent, and then recover back at 50 percent.
Unfortunately, this change also soft capped my FPS to about 37. So, yes it stabilized the client, and it was still playable, considering that original TV's refreshed at 32 FPS. I just feel that this solution isn't acceptable.
Of course, there is more. I ran the TorqueScript client against the same dedicated server and it was even less responsive then the slowed down DotNetTorque client. So on one hand I felt that I had a better client even with slowing down the interval, but on the other hand, artificially slowing down the engine seemed, well like a hack.
Having all of this research and data available to me now I believe I have found a solution which will solve the issue at hand. This solution will allow the T3D.exe and DotNetTorque each to run in their own application space. Thus, the spiking of the T3D.exe application will no longer affect the CLR runtime instance running DotNetTorque.
I have already started converting the linkage over to the new method and plan to have a new release candidate out in a couple weeks. The new release will instead use "Hooking" to couple the two applications together.
The only potential downside to the new solution is that anyone who wishes to release a game professionally will need to digitally sigh there code with a certificate from a public certificate publisher (i.e. Verisign, etc.) This will prevent Windows from considering the code unsafe and throwing security issues.
I apologize for the delay in the release, but believe me when I say I spend every waking moment debugging, testing, and writing DotNetTorque. And I refuse to release a product which I feel is not stable 100 percent of the time.
Vince
About the author
www.winterleafentertainment.com
#2
Vince
07/10/2012 (5:25 am)
Not really, like I said, if your just building a game for you and your buds, it doesn't matter. If you release a professional game then you would most likely want it signed. I plan to have more details on it soon.Vince
#3
07/10/2012 (7:57 am)
positech.co.uk/cliffsblog/2012/07/01/my-experience-with-code-signing-certificate... 
#4
The only potential downside to the new solution is that anyone who wishes to release a game professionally will need to digitally sigh there code with a certificate from a public certificate publisher (i.e. Verisign, etc.) This will prevent Windows from considering the code unsafe and throwing security issues."
well, definitively,
T3D needs more than that, since without the mods you will make that will require a certificate, i get:
C:\Users\Username\AppData\Local\Apps\2.0\ZPX4B6RM.KX9\DERLQG8A.D21\dnta..tion_11f998f695847e80_0000.0000_eb43d19577438f03\DNT FPS Demo.exe
Troyan.Win32.Jorik.IRCbot.nse
i believe you when you said on the other thread that it is safe, due to compilers inside torque, but
it already scares people away.
i heard about the torque embedded compiler that alerts the antivirus, but had never happened to me until i installed dotnet t3d.
so something must be done if you want others to play your game and feel safe.
07/10/2012 (11:48 am)
QUOTE:"The only potential downside to the new solution is that anyone who wishes to release a game professionally will need to digitally sigh there code with a certificate from a public certificate publisher (i.e. Verisign, etc.) This will prevent Windows from considering the code unsafe and throwing security issues."
well, definitively,
T3D needs more than that, since without the mods you will make that will require a certificate, i get:
C:\Users\Username\AppData\Local\Apps\2.0\ZPX4B6RM.KX9\DERLQG8A.D21\dnta..tion_11f998f695847e80_0000.0000_eb43d19577438f03\DNT FPS Demo.exe
Troyan.Win32.Jorik.IRCbot.nse
i believe you when you said on the other thread that it is safe, due to compilers inside torque, but
it already scares people away.
i heard about the torque embedded compiler that alerts the antivirus, but had never happened to me until i installed dotnet t3d.
so something must be done if you want others to play your game and feel safe.
#5
Im actually rather interested in knowing what anti-virus/build/etc you use. For all the 30+ downloads we had, youre the only person that Im aware of that complained of this.
07/10/2012 (1:38 pm)
@pilotcap232Im actually rather interested in knowing what anti-virus/build/etc you use. For all the 30+ downloads we had, youre the only person that Im aware of that complained of this.
#6
kaspersky did not noticed it when installed and ran dotnet torque,
kaspersky noticed it when one of those background pc idle disk scan it returned the virus posted above.
might wanna submit FPS Demo.exe file to virustotal for a double check since mine was deleted by kaspersky.
regards
07/10/2012 (3:11 pm)
i am using kaspersky internet security 2012 on win 7 platform.kaspersky did not noticed it when installed and ran dotnet torque,
kaspersky noticed it when one of those background pc idle disk scan it returned the virus posted above.
might wanna submit FPS Demo.exe file to virustotal for a double check since mine was deleted by kaspersky.
regards
#8
Sorry to hear about your setback. But hey, better to find out now than after you shipped the produce. Good catch and seeing your work I know you and your team will have this corrected in no time.
Also, if you can show how this typically occurs in mixed projects then it might be a good resource to show future programmers how to avoid this issue. Perhaps something on Stack Overflow. I know I would be interested in what conditions cause this issue.
07/10/2012 (11:25 pm)
@Vince,Sorry to hear about your setback. But hey, better to find out now than after you shipped the produce. Good catch and seeing your work I know you and your team will have this corrected in no time.
Also, if you can show how this typically occurs in mixed projects then it might be a good resource to show future programmers how to avoid this issue. Perhaps something on Stack Overflow. I know I would be interested in what conditions cause this issue.
#9
It seems to be only an issue when you wrap a C++ unmanaged DLL inside of a DotNet managed application or DLL.
There wasn't a great deal of information on the topic, but from other game dev forums I gleam that others were having the same problems when they tried to wrap other C++ graphics engines with DotNet.
I saw tons of different suggestions, and I tried them all. The theme to the problem appeared to have to do with threading in the application model inside of DotNet and thread control inside the managed C++ dll. Since they were both living in the same application space.
My solution to the problem is to split the two into separate application spaces, so I do not have to worry about the managed and unmanaged threads and memory spaces competing with each other.
I'm still testing the new implementation, and once I'm sure it will work I will go into more details on the work around.
Vince
07/11/2012 (4:16 am)
@Frank,It seems to be only an issue when you wrap a C++ unmanaged DLL inside of a DotNet managed application or DLL.
There wasn't a great deal of information on the topic, but from other game dev forums I gleam that others were having the same problems when they tried to wrap other C++ graphics engines with DotNet.
I saw tons of different suggestions, and I tried them all. The theme to the problem appeared to have to do with threading in the application model inside of DotNet and thread control inside the managed C++ dll. Since they were both living in the same application space.
My solution to the problem is to split the two into separate application spaces, so I do not have to worry about the managed and unmanaged threads and memory spaces competing with each other.
I'm still testing the new implementation, and once I'm sure it will work I will go into more details on the work around.
Vince
#10
07/11/2012 (10:12 am)
Are you doing multiple Application Domains or multiple processes? I've done both depending on what the problem encountered actually is.
#11
stackoverflow.com/questions/470678/wrapping-unmanaged-c-in-a-managed-wrapper
www.codeproject.com/Articles/9903/Calling-Managed-Code-from-Unmanaged-Code-and-v...
Hmmm, I just reread your post and it sounds like the managed code might be sticking its nose into the thread of the unmanaged code? Is there a way to mark a thread to be off limits so the 2 code domains do not cross that boundary?
07/11/2012 (12:14 pm)
Here are some links that 'might' help:stackoverflow.com/questions/470678/wrapping-unmanaged-c-in-a-managed-wrapper
www.codeproject.com/Articles/9903/Calling-Managed-Code-from-Unmanaged-Code-and-v...
Hmmm, I just reread your post and it sounds like the managed code might be sticking its nose into the thread of the unmanaged code? Is there a way to mark a thread to be off limits so the 2 code domains do not cross that boundary?
#12
I was running the T3D dll and the managed csharp in the same application domain. This seemed to be the crux of the problem. The reason I believe this is because the Stock T3D application would run in the same circumstance but just at 2-3 frames per second, it wouldn't crash.
So now, the new design is to have T3D and DNT run in separate processes and use some fancy fishing hooks to connect the two.
I am also almost done with the first cut, just reworking all of the externs, and I hope to be done w/ that by the weekend.
07/12/2012 (3:47 am)
@John,I was running the T3D dll and the managed csharp in the same application domain. This seemed to be the crux of the problem. The reason I believe this is because the Stock T3D application would run in the same circumstance but just at 2-3 frames per second, it wouldn't crash.
So now, the new design is to have T3D and DNT run in separate processes and use some fancy fishing hooks to connect the two.
I am also almost done with the first cut, just reworking all of the externs, and I hope to be done w/ that by the weekend.
#13
I'm using the Marshaling CSharp library to go from the CSharp to the C++, this works pretty damn decently. Once you figure out the Marshaling syntax and Managed->UnManaged type mappings you can pretty much do anything you want.
The problem has been and always will be going UnManaged->Managed. Microsoft never really came up with a way to do this. Especially, when your Managed code is not CLR aware.
Microsoft wants you to use the Com Interface or you need to write a managed C++ wrapper to your unmanaged c++. Either way to me is not really acceptable.
So, I decided to take a more to the "Metal" approach. Instead of writing a C++ CLR aware wrapper or converting T3D to a CLR aware dll, I instead choose to implement DLL hooking.
By using Far Hooking, when T3D makes calls to a stubbed C++ DLL class I wrote, I intercept the request at the system level and redirect the call back into DNT.
So it looks like:
There is a bit more to it than that, but that is the overall way it works. The business logic will have a library of 1000 or so externs available to it to directly talk to the T3D library.
In the end, the only thing programmers will have to deal with is a very simplified interface, they won't have to worry about the details of how it works, it just will.
I am spending a bit of time to insure that the solution APPEARS to be simple by hiding the complex stuff inside of a library that the developer will just include in there project.
A unexpected benefit of changing the design was that you can now debug the C++ T3D and the csharp scripts at the same time :)
There are several other unintended benifits that I didn't realize until I got into the implementation as well. But I want to confirm them before talking about them.
Vince
07/12/2012 (4:07 am)
@Frank,I'm using the Marshaling CSharp library to go from the CSharp to the C++, this works pretty damn decently. Once you figure out the Marshaling syntax and Managed->UnManaged type mappings you can pretty much do anything you want.
The problem has been and always will be going UnManaged->Managed. Microsoft never really came up with a way to do this. Especially, when your Managed code is not CLR aware.
Microsoft wants you to use the Com Interface or you need to write a managed C++ wrapper to your unmanaged c++. Either way to me is not really acceptable.
So, I decided to take a more to the "Metal" approach. Instead of writing a C++ CLR aware wrapper or converting T3D to a CLR aware dll, I instead choose to implement DLL hooking.
By using Far Hooking, when T3D makes calls to a stubbed C++ DLL class I wrote, I intercept the request at the system level and redirect the call back into DNT.
So it looks like:
DNT->Starts up DNT->Creates a new process and launches T3D.EXE DNT->Hook the Process ID and extern calls to a Stub C++ Class DLL to a CSharp delegate T3D->Calls Stub Class DLL extern (Implicit or Explicit linking) DNT->Intercepts the Extern Call DNT->Redirects Extern Call to a CSharp function BusinessLogic->Processes Extern Call DNT->Returns result T3D->Receives result to the extern call T3D->Continues on it's merry way, never once realizing that it's extern call was redirected to csharp.
There is a bit more to it than that, but that is the overall way it works. The business logic will have a library of 1000 or so externs available to it to directly talk to the T3D library.
In the end, the only thing programmers will have to deal with is a very simplified interface, they won't have to worry about the details of how it works, it just will.
I am spending a bit of time to insure that the solution APPEARS to be simple by hiding the complex stuff inside of a library that the developer will just include in there project.
A unexpected benefit of changing the design was that you can now debug the C++ T3D and the csharp scripts at the same time :)
There are several other unintended benifits that I didn't realize until I got into the implementation as well. But I want to confirm them before talking about them.
Vince
#14
07/12/2012 (9:21 am)
Are you using pInvoke to call directly into the C++ code, are you sure you have the method footprints correct? I've seen cases where this could cause a memory corruption.
#15
It is kind of interesting that this solution is similar in nature to how OBSE attaches to Oblivion. Of course there is not an API for Oblivion, but it is a launch, connect to process, and extend functionality.
So, can you share memory in a connection like this? Or is that not possible in Windows? I think you can in Linux.
07/12/2012 (1:59 pm)
Yeah, I read somewhere that the COM interface adds like 50 instruction cycles per call. That would have been bad news for a game engine. It sounds like you are approaching this very smartly. It is kind of interesting that this solution is similar in nature to how OBSE attaches to Oblivion. Of course there is not an API for Oblivion, but it is a launch, connect to process, and extend functionality.
So, can you share memory in a connection like this? Or is that not possible in Windows? I think you can in Linux.
#16
I've got the signatures down to an art, they are all computer generated based on parsing logic of the C++. So, I'm pretty positive that they aren't causing the problem.
@Frank,
Yes, you can share memory in a couple different ways between processes. In fact I do, I use IPC and SharedMemmoryMappedFiles to move data back and forth across the processes.
There is also a system wide mutex object in CSharp as well. I leverage it all in the new solution. Right now I just want to get it working, after that I will go through and optimize serializers, convert cpu intensive code to C++, etc.
Usually I take about 3 -5 passes against a codebase before I'm happy with it. The previous release canidate of DNT was reviewed line by line at least 10 to 20 times over the course of 5 months. I completely rewrote it 4 times.
Every instruction is optimized to my best ability, (and ReSharpers). It's funny how I spend a great deal of time google'ing things like the fastest implementation of string concatenation, or marshaling, etc, since DotNet has so many ways to skin a cat. Then, when that still doesn't give me the results I want, I convert to c++.
Somehow, along the way, this project has become an addiction and I won't be able to sleep until I release the first version.
07/13/2012 (9:28 pm)
@John,I've got the signatures down to an art, they are all computer generated based on parsing logic of the C++. So, I'm pretty positive that they aren't causing the problem.
@Frank,
Yes, you can share memory in a couple different ways between processes. In fact I do, I use IPC and SharedMemmoryMappedFiles to move data back and forth across the processes.
There is also a system wide mutex object in CSharp as well. I leverage it all in the new solution. Right now I just want to get it working, after that I will go through and optimize serializers, convert cpu intensive code to C++, etc.
Usually I take about 3 -5 passes against a codebase before I'm happy with it. The previous release canidate of DNT was reviewed line by line at least 10 to 20 times over the course of 5 months. I completely rewrote it 4 times.
Every instruction is optimized to my best ability, (and ReSharpers). It's funny how I spend a great deal of time google'ing things like the fastest implementation of string concatenation, or marshaling, etc, since DotNet has so many ways to skin a cat. Then, when that still doesn't give me the results I want, I convert to c++.
Somehow, along the way, this project has become an addiction and I won't be able to sleep until I release the first version.
#17
I've totally bypassed the Com layer btw, it is very slow and unreliable. I learned that about a year ago when I started this project. I've found the best way to do this stuff seems to be by straight process and memory manipulation.
Of course, if this attempt fails, I have one more trick up my sleeve (I have big sleeves). I figure if the injection method is too slow or doesn't work reliably, I will just take the mono engine and roll it into the engine.
One way or another, I WILL run CSharp inside of T3D reliably. :)
Vince
07/13/2012 (9:34 pm)
@Frank,I've totally bypassed the Com layer btw, it is very slow and unreliable. I learned that about a year ago when I started this project. I've found the best way to do this stuff seems to be by straight process and memory manipulation.
Of course, if this attempt fails, I have one more trick up my sleeve (I have big sleeves). I figure if the injection method is too slow or doesn't work reliably, I will just take the mono engine and roll it into the engine.
One way or another, I WILL run CSharp inside of T3D reliably. :)
Vince
#19
I know what you mean, I visit, review, revisit, refactor and rewrite things constantly. If you do end up hosting the CLR, you should consider also hosting the Microsoft version as well so we could choose ;-).
I am totally excited to see your work!
John
07/17/2012 (10:29 am)
@Vince,I know what you mean, I visit, review, revisit, refactor and rewrite things constantly. If you do end up hosting the CLR, you should consider also hosting the Microsoft version as well so we could choose ;-).
I am totally excited to see your work!
John
#20
So I have now implemented both:
CSharp wrapping c++ (i.e. CSharp application using the T3D.DLL as reference)
C++ wrapping CSharp (i.e. C++ program calling a C++ external library where the calls to the external library are hooked by CSharp.
Interesting enough, C++ wrapping CSharp was about 20 percent faster than the other. But, BOTH of them hit a point when the data gets to be too much that they freeze, at least now I know it is not the CSharp causing the issue.
I will let you know what I find.
Vince
07/18/2012 (4:39 am)
Well, I just finished rewriting the code base to use the new method.So I have now implemented both:
CSharp wrapping c++ (i.e. CSharp application using the T3D.DLL as reference)
C++ wrapping CSharp (i.e. C++ program calling a C++ external library where the calls to the external library are hooked by CSharp.
Interesting enough, C++ wrapping CSharp was about 20 percent faster than the other. But, BOTH of them hit a point when the data gets to be too much that they freeze, at least now I know it is not the CSharp causing the issue.
I will let you know what I find.
Vince
Torque Owner Lukas Joergensen
WinterLeaf Entertainment
From what i've seen it will just turn this:
http://img220.imageshack.us/img220/1568/unknownpublisher.png
Into this:
http://img403.imageshack.us/img403/5480/verifiedpublisher.png
Are there any other downsides besides being called an unknown publisher?