Client Visible Data Patch
by Jack Oneal · 01/05/2007 (2:02 pm) · 5 comments
Download Code File
This patch should work on all version of TGE and TSE. It has been tested on TSE MS4.
TGE's datablocks allow the server and clients to share constant data from map loading. This data is how ever very limited to preexisting engine coded features. The data is also hidden from client scripts but from very limited functions. Many scriptors have used message passing functions to transfer data in game, which works but is also very limited. Each message passing function can only pass a certain amount of data, about 1500MTU minus the size of the headers. The passing function's timing is generally in game while the game is in play, leading to lag. A good part of this data is constant from map loading and should be sent along with the datablocks. The vanilla engine makes this impossible for scriptors to do alone. The best example of this flawed design is the inventory system from tribes2, which you could expect to wait up to 30 seconds after opening the inventory before the user could interact with the inventory menu.
The obvious solution is to pass the information on the datablocks to the client scripts. This logic however is also very flawed. It will open many security holes and will send a lot of useless data. A minimal approach is required where the server scripts can decide what is sent to the clients. If the server scripts decide not to send any data, there is no loss in speed or bandwidth. The data can then be dealt with by client side scripts. These can be written by the original game authors or modders, since all the data that is sent is minimal can be shared with out a security violation. The obvious candidates for this functionality are inventory systems and map specific data. Anything that is constant during a map that the client needs can be sent safely over in a guaranteed fashion. This also removes the requirement for once time solutions for passing the data that are not as versatile or have been debugged as much. The time taken for passing the data is during mission load, when lag is not an issue and it does not affect game play.
This modification attempts to leave the smallest footprint in the code base and memory while being able to guarantee the transfer of data. In order to achieve this, all the data that is shared with the client must be constant and there must be no new datablocks created during the transfer. The transfer takes place while the client is connecting and receiving the normal datablock data. The amount of data transferred does not matter as it is held within the datablocks and is not copied into data buffers.
Each datablock from simdatablock and all of its inherited classes have this functionality. To activate the system, a single variable must be set to true. Here is an example:
The "hasClientVisibleData" must be set to true or the system will ignore this datablock. This variable must be set for each datablock that requires the data transfer. If you do not wish to use the client data system, then you will not have to pay in processing time or bandwidth. The variable name is pretty long and is not expect to have any collisions with established code bases.
Every datablock using the system must also specify a set of pattern filters to decide which data is to be sent to the clients. The patterns will be checked against every variable in the database, and if the pattern matches, the variable will be sent to the clients. In order to avoid unneeded complexity, the pattern system is quite simple. Each pattern will be checked against the prefix of each variable in a non case sensitive matter. Any characters longer than the pattern will be ignored. This is designed specifically to take advantage of how arrays are created in torque. Here is an example of how the patterns would be applied.
The system already existing inside of TGE will collapse the names of arrays into a single variable name. Here is an example.
If you wish to pass a whole array, then just set the pattern as the name of the array. The whole array will come up as true against the pattern and will be sent.
The patterns for the variables are given by an array with the name of "clientVisibleData" that has a linear order. The index of array must start at zero and not have any gaps. The system will walk the array until there is a missing element and then will stop. Here is an example of an array.
Every variable matching any of the given patterns will be sent to the client. Here is an example of a poorly written array.
The system will read the pattern index of 0 and 1 but will stop at index 2. The patterns 3 and 4 will be ignore and not sent to the client. The system is designed to be rigid to allow the fastest parsing of the array to lessen the overhead from the system.
The variables that sent to the client are not applied to in game objects, which the client scripts can not normally access. They applied as global variables, which can be modified by the client scripts as required. The name of each variable is appended to a prefix of "$db" and the variable values are unmodified. There is a limit on the variable name length and variable value, they must both be under 255 characters. This is a built lock from the engine itself that existed this patch. The system is designed to allow passing of whole arrays, and this limit should not cause a problem. All variables exceeding this limit will be truncated, but they will still be sent.
Here is an example of a full datablock and the resultant variables on the client side.
Server:
Client:
This is equivalent to
Note:
This system is designed to reduce complexity and lag. Only mark the minimal amount of data to be sent. A large amount of data will require longer load times for connecting clients. The data must be constant on the server, but the clients can modify the data as they see fit.
Parts of the code were written by Novanix.
This patch should work on all version of TGE and TSE. It has been tested on TSE MS4.
TGE's datablocks allow the server and clients to share constant data from map loading. This data is how ever very limited to preexisting engine coded features. The data is also hidden from client scripts but from very limited functions. Many scriptors have used message passing functions to transfer data in game, which works but is also very limited. Each message passing function can only pass a certain amount of data, about 1500MTU minus the size of the headers. The passing function's timing is generally in game while the game is in play, leading to lag. A good part of this data is constant from map loading and should be sent along with the datablocks. The vanilla engine makes this impossible for scriptors to do alone. The best example of this flawed design is the inventory system from tribes2, which you could expect to wait up to 30 seconds after opening the inventory before the user could interact with the inventory menu.
The obvious solution is to pass the information on the datablocks to the client scripts. This logic however is also very flawed. It will open many security holes and will send a lot of useless data. A minimal approach is required where the server scripts can decide what is sent to the clients. If the server scripts decide not to send any data, there is no loss in speed or bandwidth. The data can then be dealt with by client side scripts. These can be written by the original game authors or modders, since all the data that is sent is minimal can be shared with out a security violation. The obvious candidates for this functionality are inventory systems and map specific data. Anything that is constant during a map that the client needs can be sent safely over in a guaranteed fashion. This also removes the requirement for once time solutions for passing the data that are not as versatile or have been debugged as much. The time taken for passing the data is during mission load, when lag is not an issue and it does not affect game play.
This modification attempts to leave the smallest footprint in the code base and memory while being able to guarantee the transfer of data. In order to achieve this, all the data that is shared with the client must be constant and there must be no new datablocks created during the transfer. The transfer takes place while the client is connecting and receiving the normal datablock data. The amount of data transferred does not matter as it is held within the datablocks and is not copied into data buffers.
Each datablock from simdatablock and all of its inherited classes have this functionality. To activate the system, a single variable must be set to true. Here is an example:
datablock ParticleData(ChimneySmoke)
{
<< other variables >>
hasClientVisibleData=true;
};The "hasClientVisibleData" must be set to true or the system will ignore this datablock. This variable must be set for each datablock that requires the data transfer. If you do not wish to use the client data system, then you will not have to pay in processing time or bandwidth. The variable name is pretty long and is not expect to have any collisions with established code bases.
Every datablock using the system must also specify a set of pattern filters to decide which data is to be sent to the clients. The patterns will be checked against every variable in the database, and if the pattern matches, the variable will be sent to the clients. In order to avoid unneeded complexity, the pattern system is quite simple. Each pattern will be checked against the prefix of each variable in a non case sensitive matter. Any characters longer than the pattern will be ignored. This is designed specifically to take advantage of how arrays are created in torque. Here is an example of how the patterns would be applied.
pattern: name variable: name result: true pattern: name variable: name1 result: true pattern: name variable: namexxxxxx result: true pattern: name variable: NaMexxxxxx result: true pattern: name variable: na1mexxxxxx result: false
The system already existing inside of TGE will collapse the names of arrays into a single variable name. Here is an example.
$monkey[king]is equivalent to
$monkeyking
If you wish to pass a whole array, then just set the pattern as the name of the array. The whole array will come up as true against the pattern and will be sent.
The patterns for the variables are given by an array with the name of "clientVisibleData" that has a linear order. The index of array must start at zero and not have any gaps. The system will walk the array until there is a missing element and then will stop. Here is an example of an array.
hasClientVisibleData=true; clientVisibleData[0] = "count"; clientVisibleData[1] = "name"; clientVisibleData[2] = "type"; clientVisibleData[3] = "mod";
Every variable matching any of the given patterns will be sent to the client. Here is an example of a poorly written array.
hasClientVisibleData=true; clientVisibleData[0] = "count"; clientVisibleData[1] = "name"; clientVisibleData[3] = "type"; clientVisibleData[4] = "mod";
The system will read the pattern index of 0 and 1 but will stop at index 2. The patterns 3 and 4 will be ignore and not sent to the client. The system is designed to be rigid to allow the fastest parsing of the array to lessen the overhead from the system.
The variables that sent to the client are not applied to in game objects, which the client scripts can not normally access. They applied as global variables, which can be modified by the client scripts as required. The name of each variable is appended to a prefix of "$db" and the variable values are unmodified. There is a limit on the variable name length and variable value, they must both be under 255 characters. This is a built lock from the engine itself that existed this patch. The system is designed to allow passing of whole arrays, and this limit should not cause a problem. All variables exceeding this limit will be truncated, but they will still be sent.
Here is an example of a full datablock and the resultant variables on the client side.
Server:
datablock ParticleData(ChimneySmoke)
{
textureName = "~/data/shapes/particles/smoke";
dragCoefficient = 0.0;
gravityCoefficient = -0.2; // rises slowly
inheritedVelFactor = 0.00;
lifetimeMS = 3000;
lifetimeVarianceMS = 250;
useInvAlpha = false;
spinRandomMin = -30.0;
spinRandomMax = 30.0;
colors[0] = "0.6 0.6 0.6 0.1";
colors[1] = "0.6 0.6 0.6 0.1";
colors[2] = "0.6 0.6 0.6 0.0";
sizes[0] = 0.5;
sizes[1] = 0.75;
sizes[2] = 1.5;
times[0] = 0.0;
times[1] = 0.5;
times[2] = 1.0;
hasClientVisibleData=true;
clientVisibleData[0] = "count";
clientVisibleData[1] = "name";
clientVisibleData[2] = "type";
clientVisibleData[3] = "mod";
//List of Every Inventory Type
count = 2;
type[0] = armor;
name[0] = Player;
type[1] = vehicle;
name[1] = vehicle;
//Current Mod
mod = "core";
};Client:
$dbChimneySmokecount = 2; $dbChimneySmoketype0 = armor; $dbChimneySmokename0 = Player; $dbChimneySmoketype1 = vehicle; $dbChimneySmokename1 = vehicle; $dbChimneySmokemod = core;
This is equivalent to
$db[ChimneySmokecount] = 2; $db[ChimneySmoketype0] = armor; $db[ChimneySmokename0] = Player; $db[ChimneySmoketype1] = vehicle; $db[ChimneySmokename1] = vehicle; $db[ChimneySmokemod] = core;
Note:
This system is designed to reduce complexity and lag. Only mark the minimal amount of data to be sent. A large amount of data will require longer load times for connecting clients. The data must be constant on the server, but the clients can modify the data as they see fit.
Parts of the code were written by Novanix.
About the author
#2
01/06/2007 (8:41 am)
Awesome. This is a very clean way to pass selective data to the clients, congrats.
#3
01/15/2007 (4:27 am)
Very nice, but is there any chance we can get a text version of changes instead of a patch file for those who have heavily modified their source?
#4
01/15/2007 (9:34 pm)
Take a look at the patch format. Its pretty simple to read. It lists the name of the file that is being modified and then has the lines that are modified and those around it. The lines added are marked by + and the lines removed are marked by -.
#5
As far as I'm concern it's a great resource and we definitely want this in our game! So please help!
First of all, I'm not sure how to set patterns which seems to be essential to this resource. If it doesn't need a set in the codes how should I understand the following example? Or if it needs a set, where in the code do we have to set the followings?
And second of all, we are not clear about the lines under the "clientVisibleData=true" I couln't understand the examples. Maybe I'm too confused.
Last but not least, Where do we have to put followings in the client? Since it's global we can add it anywhere in the client side? or there is a better place to but?
If there is anything you can pitch in to help me understand little better, please help! And if it's possible it would be much appreciated if someone who understands this or using this could give me a example files if it's possible.
Many thanks a head
07/10/2007 (12:24 am)
We tired to implement this to our game but we couldn't.As far as I'm concern it's a great resource and we definitely want this in our game! So please help!
First of all, I'm not sure how to set patterns which seems to be essential to this resource. If it doesn't need a set in the codes how should I understand the following example? Or if it needs a set, where in the code do we have to set the followings?
pattern: name variable: NaMexxxxxx result: true pattern: name variable: na1mexxxxxx result: false
And second of all, we are not clear about the lines under the "clientVisibleData=true" I couln't understand the examples. Maybe I'm too confused.
hasClientVisibleData=true; clientVisibleData[0] = "count"; clientVisibleData[1] = "name"; clientVisibleData[2] = "type"; clientVisibleData[3] = "mod";
Last but not least, Where do we have to put followings in the client? Since it's global we can add it anywhere in the client side? or there is a better place to but?
$db[ChimneySmokecount] = 2; $db[ChimneySmoketype0] = armor; $db[ChimneySmokename0] = Player; $db[ChimneySmoketype1] = vehicle; $db[ChimneySmokename1] = vehicle; $db[ChimneySmokemod] = core;
If there is anything you can pitch in to help me understand little better, please help! And if it's possible it would be much appreciated if someone who understands this or using this could give me a example files if it's possible.
Many thanks a head

Torque Owner Derk Adams
I wish you had posted this a couple years ago before I invested in a shared variable system. I'm too far to rip it out and use your system, but it would have been better.
Thanks.