Client Visible Data Patch v2

Download Code File

This patch should work on all version of TGE and TSE. It has been tested on TSE MS4.

TGE's datablocks allow the server and clients to share constant data from map loading. This data is how ever very limited to preexisting engine coded features. The data is also hidden from client scripts but from very limited functions. Many scriptors have used message passing functions to transfer data in game, which works but is also very limited. Each message passing function can only pass a certain amount of data, about 1500MTU minus the size of the headers. The passing function's timing is generally in game while the game is in play, leading to lag. A good part of this data is constant from map loading and should be sent along with the datablocks. The vanilla engine makes this impossible for scriptors to do alone. The best example of this flawed design is the inventory system from tribes2, which you could expect to wait up to 30 seconds after opening the inventory before the user could interact with the inventory menu.

The obvious solution is to pass the information on the datablocks to the client scripts. This logic however is also very flawed. It will open many security holes and will send a lot of useless data. A minimal approach is required where the server scripts can decide what is sent to the clients. If the server scripts decide not to send any data, there is no loss in speed or bandwidth. The data can then be dealt with by client side scripts. These can be written by the original game authors or modders, since all the data that is sent is minimal can be shared with out a security violation. The obvious candidates for this functionality are inventory systems and map specific data. Anything that is constant during a map that the client needs can be sent safely over in a guaranteed fashion. This also removes the requirement for once time solutions for passing the data that are not as versatile or have been debugged as much. The time taken for passing the data is during mission load, when lag is not an issue and it does not affect game play.

This modification attempts to leave the smallest footprint in the code base and memory while being able to guarantee the transfer of data. In order to achieve this, all the data that is shared with the client must be constant and there must be no new datablocks created during the transfer. The transfer takes place while the client is connecting and receiving the normal datablock data. The amount of data transferred does not matter as it is held within the datablocks and is not copied into data buffers.

Each datablock from simdatablock and all of its inherited classes have this functionality. To activate the system, a single variable must be set to true. Here is an example:

datablock ParticleData(ChimneySmoke)
{
    << other variables >>
     hasClientVisibleData=true;
};

The "hasClientVisibleData" must be set to true or the system will ignore this datablock. This variable must be set for each datablock that requires the data transfer. If you do not wish to use the client data system, then you will not have to pay in processing time or bandwidth. The variable name is pretty long and is not expect to have any collisions with established code bases.

Every datablock using the system must also specify a set of pattern filters to decide which data is to be sent to the clients. The patterns will be checked against every variable in the database, and if the pattern matches, the variable will be sent to the clients. In order to avoid unneeded complexity, the pattern system is quite simple. Each pattern will be checked against the prefix of each variable in a non case sensitive matter. Any characters longer than the pattern will be ignored. This is designed specifically to take advantage of how arrays are created in torque. Here is an example of how the patterns would be applied.

pattern: name
variable: name
result: true

pattern: name
variable: name1
result: true

pattern: name
variable: namexxxxxx
result: true 

pattern: name
variable: NaMexxxxxx
result: true 

pattern: name
variable: na1mexxxxxx
result: false

The system already existing inside of TGE will collapse the names of arrays into a single variable name. Here is an example.

$monkey[king]

is equivalent to

$monkeyking

If you wish to pass a whole array, then just set the pattern as the name of the array. The whole array will come up as true against the pattern and will be sent.

The patterns for the variables are given by an array with the name of "clientVisibleData" that has a linear order. The index of array must start at zero and not have any gaps. The system will walk the array until there is a missing element and then will stop. Here is an example of an array.

hasClientVisibleData=true; 
clientVisibleData[0] = "count";
clientVisibleData[1] = "name";
clientVisibleData[2] = "type";
clientVisibleData[3] = "mod";

Every variable matching any of the given patterns will be sent to the client. Here is an example of a poorly written array.

hasClientVisibleData=true; 
clientVisibleData[0] = "count";
clientVisibleData[1] = "name";
clientVisibleData[3] = "type";
clientVisibleData[4] = "mod";

The system will read the pattern index of 0 and 1 but will stop at index 2. The patterns 3 and 4 will be ignore and not sent to the client. The system is designed to be rigid to allow the fastest parsing of the array to lessen the overhead from the system.

The variables that sent to the client are not applied to in game objects, which the client scripts can not normally access. They applied as global variables, which can be modified by the client scripts as required. The name of each variable is appended to a prefix of "$db" and the variable values are unmodified. There is a limit on the variable name length and variable value, they must both be under 255 characters. This is a built lock from the engine itself that existed this patch. The system is designed to allow passing of whole arrays, and this limit should not cause a problem. All variables exceeding this limit will be truncated, but they will still be sent.

Here is an example of a full datablock and the resultant variables on the client side.

Server:

datablock ParticleData(ChimneySmoke)
{
   textureName          = "~/data/shapes/particles/smoke";
   dragCoefficient     = 0.0;
   gravityCoefficient   = -0.2;   // rises slowly
   inheritedVelFactor   = 0.00;
   lifetimeMS           = 3000;
   lifetimeVarianceMS   = 250;
   useInvAlpha = false;
   spinRandomMin = -30.0;
   spinRandomMax = 30.0;

   colors[0]     = "0.6 0.6 0.6 0.1";
   colors[1]     = "0.6 0.6 0.6 0.1";
   colors[2]     = "0.6 0.6 0.6 0.0";

   sizes[0]      = 0.5;
   sizes[1]      = 0.75;
   sizes[2]      = 1.5;

   times[0]      = 0.0;
   times[1]      = 0.5;
   times[2]      = 1.0;
   
    hasClientVisibleData=true; 
    clientVisibleData[0] = "count";
    clientVisibleData[1] = "name";
    clientVisibleData[2] = "type";
    clientVisibleData[3] = "mod";

    //List of Every Inventory Type
    count = 2;
    type[0] = armor;
    name[0] = Player;
    type[1] = vehicle;
    name[1] = vehicle;

    //Current Mod
    mod = "core";
};

Client:

$dbChimneySmokecount = 2;
$dbChimneySmoketype0  = armor;
$dbChimneySmokename0 = Player;
$dbChimneySmoketype1 = vehicle;
$dbChimneySmokename1 = vehicle;
$dbChimneySmokemod = core;

This is equivalent to

$db[ChimneySmokecount] = 2;
$db[ChimneySmoketype0]  = armor;
$db[ChimneySmokename0] = Player;
$db[ChimneySmoketype1] = vehicle;
$db[ChimneySmokename1] = vehicle;
$db[ChimneySmokemod] = core;

In order to activate the system, a hook must be called after the map is loaded and before clients load. Call

buildClientVisibleData();

in order to update the visible data array. Since most engines have a modified system, this has to be found by the coder. In the example project, a good location to put this call is in ./example/common/server/missionLoad.cs around line 35.

Before:

// Extract mission info from the mission file,
   // including the display name and stuff to send
   // to the client.
   buildLoadInfo( %missionName );

   // Download mission info to the clients
   %count = ClientGroup.getCount();
   for( %cl = 0; %cl < %count; %cl++ ) {
      %client = ClientGroup.getObject( %cl );
      if (!%client.isAIControlled())
         sendLoadInfoToClient(%client);
   }

After

// Extract mission info from the mission file,
   // including the display name and stuff to send
   // to the client.
   buildLoadInfo( %missionName );

   buildClientVisibleData();

   // Download mission info to the clients
   %count = ClientGroup.getCount();
   for( %cl = 0; %cl < %count; %cl++ ) {
      %client = ClientGroup.getObject( %cl );
      if (!%client.isAIControlled())
         sendLoadInfoToClient(%client);
   }

Note:
This system is designed to reduce complexity and lag. Only mark the minimal amount of data to be sent. A large amount of data will require longer load times for connecting clients. The data must be constant on the server, but the clients can modify the data as they see fit.

Parts of the code was written by Novanix.