Upfront Datablock Loading

Download Code File

Speed loading/persistance ho!

First, a video of speedloading in action, or the effect of it anyway, VERY fast mission loading.
Speedloading in Cubekind

Second, a video of speedloading in action in stock ArcaneFX ComboPack, of which the number of datablocks is a well known quantity
Speedloading in ArcaneFX


I have often noted that I do not change datablocks among server instances, and my favorite kinds of projects to work on are the kinds that need you to jump from server to server, like RPGs and Cubekind. So the datablock loading process has always bugged me. Especially now that I've got lots of other things working like server instancing and multiple varieties of server to server zoning working.

I am also an ArcaneFX addict, and it needs some heavy datablock definitions to get the most bang and variety. So I usually end up with near 4000 datablocks just in a prototype/Alpha, and could probably double that number with some serious content work.

To wit, I have a need to not reload datablocks on mission loads. I want them to stick around and skip that part of the mission loading process. Furthermore, I want the datablocks to load up while credits/titles are loading, as this is an acceptable place for a front load.

The nature of the process is to create and connect to a local server in an optimized way. A mission is not loaded, but the server does its normal exec process and loads all its datablocks. The client then connects locally to the server and disconnects when datablock transmission is complete, setting a flag that says it has the datablocks loaded. From there on in, wether connecting to a local server or remote server, datablocks are not transmitted.

Assumptions:
1) Datablocks are not going to change between server instances ie they all have the same execs and order of execs.
2) Server code/assets goes out with the distribution, making the local server possible

If I did not want to distribute server code I would look further into pulling code out of Jeff Faust's ArcaneFX datablock caching scheme to create a binary datablock cache file and load that instead on startup. That would be really sweet indeed. Nevertheless, I've had troubles with hashes etc and this process has been working very nicely.

Let's begin! Anywhere I use the word game as part of a path, replace with your own... starter.fps, arcane.fx, whatever. It just means the main game folder.

Instructions:
The attached script file can be placed in your game/client/scripts folder and contains NEW functions which are also included here.

in game/client/init.cs

near the end of function initClient()

above:

// Default player key bindings
   exec("./scripts/default.bind.cs");

add:

exec("./scripts/preloader.cs");

This takes care of loading up the preloader functions for us as soon as the client scripts are loaded. I did it this way so that if there were any client datablocks created, hopefully they had a chance to run and sound profiles get setup, etc. Early on I experimented with having the load happen at the top of initClient, but results were not consistent.

What happens now is that the loadLoaderServer function gets called, and that creates a tiny server with no mission file and the client will connect to it when it's done loading.

function doLoaderServer()
{
      createLoaderServer("Loader");
      %conn = new GameConnection(LoaderServerConnection);
      RootGroup.add(LoaderServerConnection);
      %conn.LoaderServer = true;
      %conn.setConnectArgs("LoaderConnect","LoaderConnect","LoaderConnect");
      %conn.setJoinPassword($Client::Password);
      %conn.connectLocal();
}

The client connect/disconnect process is tightly bound, so we are going to do several things to make it easier to tell later on in other parts of code that this connection is a special connection only used to load stuff up. We did this by naming it something besided ServerConnection, attaching a dynamic field named LoaderServer onto the object, and also setting its connect args to "LoaderConnect" in all the available places. The connect args are needed because when the server gets the connect request, it can know that the connection is a special type and must be treated specially.

The createLoaderServer function is really just a specialized version of createServer().
It sets a flag that will be needed later ($LoaderServer) which can be used while execing, etc to skip anything that doesn't need to happen
in this special mini server. It also says $missionRunning = true, so that the normal datablock load sequence can take place.

function createLoaderServer()
{
   echo("*********Datablock Preload*********");
   $missionSequence = 0;
   $LoaderServer = true;
   allowConnections(false);
   
   $ServerGroup = new SimGroup(ServerGroup);

   //This loads up all the datablocks, as normal   
   onServerCreated();
   
   $MissionRunning = true;
}

Now that the server is loaded, the client continues its local connection process and connects locally to the running server.
I frequently use extra arguments in the connection process to do special things in the load order, so I add a few extra arguments (which the engine already supports) in addition to the normal %client and %name arguments.

Now, let's shortcircuit the normal loading process, all we want is a server-provided datablock stream after all.

In common/server/clientConnection.cs, change the function declaration to:
function GameConnection::onConnect( %client, %name, %extra,%extra2)

and add in, near the top:

if(%extra $= "LoaderConnect")
   {
      echo("LoaderConnect");
      %client.LoaderConnect = true;
      %client.transmitDataBlocks($missionSequence);
      return;
   }

This has the effect of flagging the connection as a loaderConnect type connection, because we might need to know this later to do special handling on it or skip other processing. We also start transmitting datablocks instead of telling the connection to load the mission.

This causes function GameConnection::loadMission in game/server/commands.cs to run, the last line of which is:
commandToClient(%this, 'MissionStartPhase1', $missionSequence, $Server::MissionFile, MissionGroup.musicTrack, %cache_crc);

From here, datablock transmission is handled as normal. When the client has all the datablocks, we don't have need of the server anymore, so we destroy it and tell the client to drop.

In common/server/missionDownload.cs, at the top of:
function GameConnection::onDataBlocksDone( %this, %missionSequence )

add:

//Special handling, kill the conn, destroy the server
   if($LoaderServer == true)
   {
      echo("Datablock loading complete, loader server going bye bye");
      commandToClient(%this, 'DatablocksLoaded');
      destroyLoaderServer();
      return;
   }

The client gets told that the datablocks are loaded, and it's their job to then record that and delete the connection.
The $DataBlocksLoaded global is used to determine where in the preload process we are, as this should happen as the first title screens are loading. The $SkipDataBlocks global is almost the same thing, but has its own use on future mission load sequences.

function ClientCmdDatablocksLoaded()
{
   $SkipDataBlocks = true;
   $DataBlocksLoaded = true;
   LoaderServerConnection.delete();
   echo("Datablocks loaded");
}

After the command is sent to the client, the server destroys itself. This causes the server to suicide, and we don't want to lose all the datablocks we have loaded up, so we have a special version of destroyServer called destroyLoaderServer which skips the deleting of datablocks and purging of resources.

function destroyLoaderServer()
{
   $Server::ServerType = "";
   allowConnections(false);
   stopHeartbeat();
   $missionRunning = false;
   
   if (isObject($ServerGroup))
      $ServerGroup.delete();
 
   $Server::GuidList = "";
}

How do we kick this all off?
Well, I wanted to impact as few existing files as possible, so I chose to modify the loading Guis process.

in game/client/ui/StartupGui.Gui, in function loadStartup, add these lines at the top of the function

$DataBlocksLoaded = false;
   doLoaderServer();

And, in function checkStartupDone,
change:

if (StartupGui.done)

to:

Canvas.repaint;
if ($DataBlocksLoaded)

When the gui opens up, the preload process begins, and this will make sure the datablock loading is done during the first credit screen. I also chose to add a progress bar and hook into the normal datablock progression code, but I leave that as an exercise for the reader.

Now we have datablocks loaded up, before we hit the main menu. This takes about 7-10 seconds for me, with all arcaneFX spells loaded and an additional 600 datablocks, putting my number of datablocks near 4000.

The next part of this is to SKIP the datablocks on future mission loads. We used a variable named $SkipDatablocks to set this state, after all our datablocks were loaded.


In game/client/scripts/missionDownload.cs:
function clientCmdMissionStartPhase1
after the echos:

if ($skipdatablocks==true)
  {
     echo("<<<< skip datablocks >>>>");
     // skip datablock transmission and initiate a cache load
      onMissionDownloadPhase1(%missionName, %musicTrack, "PRELOADING DATABLOCKS");
      commandToServer('MissionStartPhase1Ack_UseExistingDatablocks', %seq);
      return;
   }

This is called in the normal mission load process. From here, the rest of the custom loading functions take over. They are all contained in preloader.cs.

function serverCmdMissionStartPhase1Ack_UseExistingDatablocks(%client, %seq)
{
  echo("    <<<< skipping datablock transmission >>>>");

  // Make sure to ignore calls from a previous mission load
  if (%seq != $missionSequence || !$MissionRunning)
    return;
  if (%client.currentPhase != 0)
    return;
  %client.currentPhase = 1;

  // Start with the CRC
  %client.setMissionCRC( $missionCRC );
  %client.onBeginDatablockExistingLoad($missionSequence);
}

function GameConnection::onBeginDatablockExistingLoad( %this, %missionSequence )
{
   // Make sure to ignore calls from a previous mission load
   if (%missionSequence != $missionSequence)
      return;
   if (%this.currentPhase != 1)
      return;
   %this.currentPhase = 1.5;
   commandToClient(%this, 'MissionStartPhase1_LoadExisting', $missionSequence, $Server::MissionFile);
}

function clientCmdMissionStartPhase1_LoadExisting(%seq, %missionName)
{
  if ($skipdatablocks==true)
  {
    echo("<<<< Skipping Datablocks");
    schedule(10, 0, "updateLoadExistingDatablockProgress", %seq, %missionName);
  }
}

function updateLoadExistingDatablockProgress(%seq, %missionName)
{
   echo("<<<< Finished Skipping Datablocks >>>>");
   clientCmdMissionStartPhase2(%seq,%missionName);
}

That really takes care of things from the script side of things. The one last thing to keep in mind is that the client's disconnect() function also calls destroyServer(). destroyServer() has a call in it to deleteDatablocks().

I am using custom disconnect functionality, so if you are using stock disconnect, you will want to add a conditional over it, like this:
if(!$datablocksloaded)
deleteDataBlocks();

For the engine side, use a different transmitDatablocks function. There are some good alternatives in this resource:
http://www.garagegames.com/index.php?sec=mg&mod=resource&page=view&qid=12879

I used the versions given by Daniel Eden with the changes noted by Ed Zavada Posted: (Mar 17, 2008 at 23:47). Very nice resource!!

Speed loading ho!! Please post improvements, etc.