The pain of T3D and Multithreading

Let`s talk about threading.

T3D really does not like to thread. Beyond animation sequences and such T3D is a single thread application. This is truly a shame.

So recently I had a very unique problem. I wanted to run a basic NetEvent message router in the background of the server and I wanted the message router to use NetEvents. Basically, the server could drop message data into a Queue as it wanted to and a background thread would watch the queue and send them out as NetEvents to the connected clients.

There were some caveats:

-The messaging process would only send each connection the NetEvent once.
-The NetEvent had to be sent in the order they were put into the Queue.
-Each connection could be at a different point in the reception of the queued NetEvents.

So, I first used a Tickable class, but the 32ms delay was just too slow. One message per connection per client just didn`t give the smoothness I needed for the terrain editor.

So I then tried using the good ole fashion Sim::PostEvent with a SimTime() +1. Still, sending one message per connection per 1ms per connection was still too slow. I needed as near to instantaneous as possible, and this didn`t provide it.

I must have gone around this for a few hours until I decided to give threading a try. So I set up all the threading requirements and spun off a thread that cycled through each of the connections and it would continuously send NetEvent`s to the clients.

Then I found out that GameConnection::postNetEvent wasn`t thread safe. So, I started looking at the class and attempted to make it thread safe. To put it mildly it was a fail.

After a bit of digging, I realized that Sim::PostEvent was thread safe. So I wrapped my Gameconnection::postNetEvent inside a SimEvent and posted the SimEvent from the thread.

I ended up making a few other minor changes but for the most part it was sending the messages at an acceptable rate.

We then started loading in multiple clients, had people drop, etc. The result was simple, the server crashed. Since I was cycling through the game connections rapidly and out of sync of the main thread of T3D, I encountered a situation of a connection being in the process of being deleted or created during the background thread loop. Or while I was holding onto a Connection pointer, T3D would delete the connection. Either way I had a problem.

So I started looking at the Weak and Strong reference part of the engine. It provided some of the functionality I needed but it just wasn`t tailored for exactly what I wanted. So I took a day to sit back and think and on my drive home the next day it hit me.

Since all SimObjects (GameConnections included) need to register and unregister themselves with the SimDictionary, let`s put a locking mechanism there. When my message router processed a loop of the connections, it would put a lock on any request for Adding or Removing GameConnections pointers from the dictionary, when the loop finished it would release the lock.

Of course at this point I ended up having 3 separate message routers, and it appeared that they always managed to have a lock on GameConnection pointers, so the engine would just hang since T3D could never get a lock on a GameConnection pointer.

Thus was born a priority lock (Whether I invented this or not, it came from my bag of Black Magic). A priority lock was a lock which would block any locks from occurring of a lesser priority. So, let`s say we have two locks:

-Message Router Lock (Low Priority)
-SimDictionary Lock (High Priority)

Multiple Message Router Lock locks are allowed, they don`t block each other, and they only block SimDictionary Lock`s.

SimDictionary Lock`s always BLOCK each other and Message Router Lock.

Each Message Router Lock increments a lock counter by 1, and the release of the lock decrease the lock counter by 1.

When a SimDictionary Lock occurs, it blocks and makes all new Message Router Lock`s wait. The SimDictionary Lock then waits until the lock counter hits zero. Once the lock counter hits zero, it returns control back to the SimDictionary. After the insert or delete is finished, the SimDictionary Lock releases its lock thus unblocking all the queued Message Router Lock`s.

Mind you, this doesn`t lock individual objects, but instead a Type of object, in my case it was GameConnections, with a little bit of work I could easily template the class and set it up to provide locking for all objects by type or individual, but to be honest, I`m not that ambitious. <Grin>

Long story short, my NetEvent message router now works like a champ sending messages almost as fast as it receives them!

Locking code looked like:

inline void LockGameConnections()
	   {
	   while (true)
		   {
		   Mutex::lockMutex(FlagMutex);
		   if (!mLockRequested)
			   break;
		   Mutex::unlockMutex(FlagMutex);
		   Platform::sleep(1);
		   }
	   Mutex::unlockMutex(FlagMutex);
	   Mutex::lockMutex(ReadMutex);
	   mLockCounter++;
	   Mutex::unlockMutex(ReadMutex);
	   };

   inline void UnLockGameConnections()
	   {
	   Mutex::lockMutex(ReadMutex);
	   mLockCounter--;
	   Mutex::unlockMutex(ReadMutex);
	   };
   inline void RequestPriorityLock()
	   {
	   Mutex::lockMutex(FlagMutex);
	   mLockRequested=true;
	   Mutex::unlockMutex(FlagMutex);
	   while (mLockCounter>0)
		   {Platform::sleep(1);}
	   };
   inline void ReleasePriorityLock()
	   {
	   Mutex::lockMutex(FlagMutex);
	   mLockRequested=false;
	   Mutex::unlockMutex(FlagMutex);
	   }

Vince
Winterleaf Entertainment L.L.C.