Programmers DTS Introduction (At least... its a start)

Hello everyone.

Well... I was going to post 32k of notes I've taken related to DTS and the engine, but I can only post 4k, it wasn't enough space really to even wet your appetite, so I'll summarize and ask where I can post the rest?

I've been searching around for info on the DTS file format. I wanted to know how the bits are stored on disk, where they are read into
the engine, how these are converted to data structures, etc.

I did find some information in the forums/docs, but not exactly what I was after. Sorry if I missed the magic post...

So, I've written down what I've discovered so far through the source. I've used the remaing post space to give you as sample/teaser. How to I make the rest available?

- Kevin

The DTS FORMAT:

The DTS format is ... creative. Rather than dump memory directly to disk
one object at a time, the file is divided into several sections and
the payload is scattered across the sections based on the data type of
the object to be stored. This means that there is a 32-bit section, a
16-bit section and an 8-bit section for the primary payload. As a data
structure is stored, it will be written, distributed across these
sections according to the data type of the member being written.

For example, let's say the following data structure needed to be written:

struct mytype {
int x;
int y;
int z;
short r; // radius
float s; // scale
unsigned char i; // index
};

The data would be arranged in the file as follows:

.../----32-bit---------/----16-bit----/----8-bit----/...
xxxxyyyyzzzzssss!!!!rr!! i!

Note, this is 32-bits of the ints x,y, and z followed by 32-bits of
the float, s. Notice that the 16-bit, r, was written into the 16-bit
section and the 8-bit, i was written to the 8-bit section. The
exclamation point symbols represent a "marker" which ensure that the
data structures, though distributed are actually aligning with one
another when retrieved. These are integer values that are incremented
for each successive data structure written to the file. Notice that
the file format limits the number of checkpoints possible to 256
because of the distribution of the checkpoint into the 8-bit data.

Also of note, each section is 4-byte aligned, though all data is
packed into the front of the section. The format appears to require
that the number of items stored, by count, in each section is
multiplied by 4 to accomplish this alignment. This seems wasteful, in
order to store n, 8-bit, values, you would have to allocate space for
(n*4) items! I'm still investigating, it's hard to imagine this was
desired. It may simply be legacy.

Preceding this data is a small header that gives the version of the
DTS file and the offsets of the 32, 16 and 8 bit
sections. Interestingly, these offsets are given as counts and the
last is implied:

Header:
Version: 24
totalSize: 491
offset16: 403
offset8:464

The "totalSize" value is the total number of 32-bit values in the
distributed payload. In this case 491 * 4-bytes == 1964. The 32-bit
data is the 403, 32-bit values following the header. The 16-bit data
begins at byte (403 * 4 == 1612), and is valid through the start of
the 8-bit data at byte (464 * 4 == 1856). So here's the range of data
for the sample header that will need to be read after the header:

TYPE OFFSET
32-bit [ 0 - 1611]
16-bit [1612 - 1855]
8-bit [1856 - 1965]

I used a tool I wrote, called dtsdump, to dump the healthKit.dts file
from the starter.fps/data/shapes/items/ folder. The header is stored
as a straight dump:

00000000: 1800 0000 eb01 0000 9301 0000 d001 0000 ................
00000010: 0400 0000 0200 0000 0000 0000 0100 0000 ................

Adjusting for endian:
0x00000018 == 24, 0x000001eb == 491, 0x00000193 == 403, 0x000001d0 == 464

... ACK! NO MORE SPACE! ...