Reducing level load time

#41 I guess I am looking for a document that describes the traditional short cut method Tim McClarren described above. I am guessing that this is the same as "short circuiting the init phase" that Ben talked about.

I inherited a game from another programmer, and I know quite a bit of time is being spent with sending datablocks and ghosted object info to the "client" from the "server", even though this is a single player game. I understand why you would do this architechure for client server games , I just dont see any information on how to declare datablocks and object descriptions so that they do not need to be sent from the server, and I can save this time during the load.

I am guessing this might be simple to do, anyone point me in the right direction? Any sample code I can look at or documents would be greatly appreciated.

#42 The "short circuit" method I was talking about was the one posted by Cameron Tofer on 7/4/2005 -- there's not really a document, but the method has been discussed in this thread and possibly one or two others.

Incidentally, I notice that there was no followup regarding the missing textures in particle effects bug that Tom Spilman described in his 9/6/2005 post, and this is *very* curious to me, because I've implemented the system that I've described, which is very different, and it *too* exhibits this bug on occasion (particularly, standalone). If someone has investigated this, I'd be curious to hear about it!

Cameron's load-optimization changes all run single-process on one machine for speeding up a single-player game load time.

The changes I describe actually implement an approach for speeding up load times for both a single-process (standalone game) or a client-server game by caching the bitstream that is generated for a given server for a given mission (they are made somewhat non-trivial by the CRC check, which, if you don't really care about, you could implement a much simpler method which would not involve fixing all of the rest of the places in the engine that are sending garbage values over the wire).

#43 Ok thank you Tim for clarifying, I will implement the speed up that Cameron Tofer described, and cross my fingers I dont get the missing particle effects bug that Tom Spilman and you talk about seeing.

#44 Guessing no one has solved the missing textures in particles / effects mystery?

#45 Well, I did implement the changes to the prefs file that Cameron Aycock mentioned, and the C++ code changes to TransmitDataBlocks method that Cameron Tofer listed in this thread and I had no problems. I was relieved at this, though I did keep the datablock size to 400 for safety. I got quite a nice speed up on the loading process.

#46 I suggest, doing a new object class, which is not created at server and not ghosted to client. This new object type is created in the server and client separately and put in the appropriate object list.

Create a file which have a list of static objects for the level and put this list at the client and at the server.
At level load time, client and server creates objects internally using this file. As long as you keep the files on the client and server same, there will be no problems and the speed is extraordinary.

We are using this in an MMO which uses Torque and handling very high static object counts. Level load times are superb.

We are only creating dynamic objects in the server and ghosting them to clients like players.

#47 Cem, could you provide some sample code of how this would be done?

#48 Cem, this is essentially the same as my approach of creating a cached version of the datablocks and the ghost always objects, as I suggested.

Only, the cache approach has the added advantage that you can change the mission after the fact and the server can determine if a client has an out-of-date mission and tell the client to download again and cache the updated version.

We use this approach in our MMO.

#49 Transferring the large mission/static objects file over UDP takes a long time. We are using RTPatch for our client updates.

#50 We just started using an auto-patch feature of NSIS.
we'll see how it goes.

#51 Okay how much would one charge for a working approach to this problem. I have tried all the approaches here that have code but most are for single player and I am looking for a multiplayer solution. Seriously I really think that someone at GG should step up and fix this. I think everyone would benefit from this code. Be it caching the datablocks or something else. This would be a giant leap for all of us indie developers.

#52 @Eric - Actually this is exactly the sort of thing a community member should take on and resource. It's a focused and well understood problem with a few different publicly discussed solutions. Someone with the need and time to spend just needs to "man up" and get it done.

This thread started almost 3 years ago... someone fix this already. I dare you! ;)

#53 It's not exactly a resource, but it's all in one place at least. I've checked this code out on local and dedicated servers, in demos and out, and it all seems to work. There may be a few issues here and there, but I haven't seen any yet. As always, use the code at your own risk. I'm not supporting it in any way. If you have problems implementing it or want questions answered or whatever, too bad.

Essentially these changes amount to little more than short-circuiting both datablock transmission and the transmission of scopealways objects so as to avoid the chokepoint imposed by networking packet sizes and the like. This will greatly decrease the amount of time required for transmitting datablocks and static objects (such as the terrain and so on) to a local client.

For my money, I'd also recommending not re-sending datablocks on subsequent mission loads (which will speed up both singleplayer and multiplayer games), but I haven't the time or the inclination to write up how to do so. It's pretty simple to figure out if you really want to do it.

Similarly it might be an idea to implement some small editor changes to automagically export dynamic objects into a separate "gameplay object" file (precluding the need for relighting the mission from scratch each and every time you move an item that has no effect on lighting).

In conjunction with such a system and with a few minor modifications to the netobject class and the scripts you can also set the engine up to execute the static mission file locally on the client -- which will vastly reduce the time requirements of ghosting highly-populated missions to each client (especially in multiplayer games). Simple CRC code can be used to ensure the client and server files are identical. Oh, and you can always stream the clients any static mission file they don't have if you need to (at about the speed it would take to load all the objects normally on a server). You have to be a bit careful about this, however, as certain static objects (such as the terrain) need to have valid ghosts for certain functionality in the game.

One last thing... I'm not 100% sure on the "particles not showing up" issue. I've never seen it happen, but I've made a fair bunch of changes around the place so that may be why. If you can come up with a reliable method of replication let someone know. =)

#54 Had to do a multi-part post because of the silly post length limits... So here we go, step-by-step:

First, around line 1230 of game/gameConnection.cc, replace the transmitDataBlocks console method with:

ConsoleMethod( GameConnection, transmitDataBlocks, void, 3, 3, "(int sequence)")
{
    // Set the datablock sequence.
    object->setDataBlockSequence(dAtoi(argv[2]));

    // Store a pointer to the datablock group.
    SimDataBlockGroup* pGroup = Sim::getDataBlockGroup();

    // Determine the size of the datablock group.
    const U32 iCount = pGroup->size();

    // If this is the local client...
    if (GameConnection::getLocalClientConnection() == object)
    {
        // Set up a pointer to the datablock.
        SimDataBlock* pDataBlock = 0;

        // Iterate through all the datablocks...
        for (U32 i = 0; i < iCount; i++)
        {
            // Get a pointer to the datablock in question...
            pDataBlock = (SimDataBlock*)(*pGroup)[i];

            // Set the client's new modified key.
            object->setMaxDataBlockModifiedKey(pDataBlock->getModifiedKey());

            // Set up a buffer for the datablock send.
            U8 iBuffer[4096];
            BitStream mStream(iBuffer, 4096);

            // Pack the datablock stream.
            pDataBlock->packData(&mStream);

            // Set the stream position back to zero.
            mStream.setPosition(0);

            // Unpack the datablock stream.
            pDataBlock->unpackData(&mStream);

            // Call the console function to set the number of blocks to be sent.
            Con::executef(3, "onDataBlockObjectReceived", Con::getIntArg(i), Con::getIntArg(iCount));

            // Preload the datablock on the dummy client.
            pDataBlock->preload(false, NetConnection::getErrorBuffer());
        }

        // Get the last datablock (if any)...
        if (pDataBlock)
        {
            // Ensure the datablock modified key is set.
            object->setDataBlockModifiedKey(object->getMaxDataBlockModifiedKey());

            // Ensure that the client knows that the datablock send is done...
            object->sendConnectionMessage(GameConnection::DataBlocksDone, object->getDataBlockSequence());
        }
    } 
    else
    {
        // Otherwise, store the current datablock modified key.
        const S32 iKey = object->getDataBlockModifiedKey();

        // Iterate through the datablock group...
        U32 i = 0;
        for (; i < iCount; i++)
        {
            // If the datablock's modified key has already been set, break out of the loop...
            if (((SimDataBlock*)(*pGroup)[i])->getModifiedKey() > iKey)
            {
                break;
            }

            // If this is the last datablock in the group...
            if (i == iCount)
            {
                // Ensure that the client knows that the datablock send is done...
                object->sendConnectionMessage(GameConnection::DataBlocksDone, object->getDataBlockSequence());

                // Then exit out since nothing else needs to be done.
                return;
            }
        }

        // Set the maximum datablock modified key value.
        object->setMaxDataBlockModifiedKey(iKey);

        // Get the minimum number of datablocks...
        const U32 iMax = getMin(i + DataBlockQueueCount, iCount);

        // Iterate through the remaining datablocks...
        for (;i < iMax; i++)
        {
            // Get a pointer to the datablock in question...
            SimDataBlock* pDataBlock = (SimDataBlock*)(*pGroup)[i];

            // Post the datablock event to the client.
            object->postNetEvent(new SimDataBlockEvent(pDataBlock, i, iCount, object->getDataBlockSequence()));
        }
    }
}

In the same file, at line 1146 or thereabouts, find the following:

// object failed to load, let's see if it had any missing files

         if(!ResourceManager->getMissingFileList(mMissingFileList))

... and replace it with ...

// object failed to load, let's see if it had any missing files

         if(isLocalConnection() || !ResourceManager->getMissingFileList(mMissingFileList))

#55 Then in sim/netGhost.cc, around line 958, find this code in NetConnection::loadNextGhostAlwaysObject:

// object failed to load, let's see if it had any missing files
         if(!ResourceManager->getMissingFileList(mMissingFileList))

... and replace it with ...

// object failed to load, let's see if it had any missing files
         if(isLocalConnection() || !ResourceManager->getMissingFileList(mMissingFileList))

In the same file around line 828 in the NetConnection::activateGhosting method, remove the following code:

sendConnectionMessage(GhostAlwaysStarting, mGhostingSequence, ghostAlwaysSet->size());
   for(j = mGhostZeroUpdateIndex - 1; j >= 0; j--)
   {
      AssertFatal((mGhostArray[j]->flags & GhostInfo::ScopeAlways) != 0, "Non-scope always in the scope always list.")

      // we may end up resending state here, but at least initial state
      // will not be resent.
      mGhostArray[j]->updateMask = 0;
      ghostPushToZero(mGhostArray[j]);
      mGhostArray[j]->flags &= ~GhostInfo::NotYetGhosted;
      mGhostArray[j]->flags |= GhostInfo::ScopedEvent;

      postNetEvent(new GhostAlwaysObjectEvent(mGhostArray[j]->obj, mGhostArray[j]->index));
   }
   sendConnectionMessage(GhostAlwaysDone, mGhostingSequence);

... and replace it with:

// Send the initial ghosting connection message.
    sendConnectionMessage(GhostAlwaysStarting, mGhostingSequence, ghostAlwaysSet->size());

    // If this is the connection to the local client...
    if (getLocalClientConnection() == this)
    {
        // Get a pointer to the local client.
        NetConnection* pClient = NetConnection::getConnectionToServer();

        // Iterate through the scope always objects...
        for (j = mGhostZeroUpdateIndex - 1; j >= 0; j--)
        {
            AssertFatal((mGhostArray[j]->flags & GhostInfo::ScopeAlways) != 0, "NetConnection::activateGhosting:  Non-scope always in the scope always list.")

            // Clear the ghost update mask and flags appropriately.
            mGhostArray[j]->updateMask = 0;
            ghostPushToZero(mGhostArray[j]);
            mGhostArray[j]->flags &= ~GhostInfo::NotYetGhosted;
            mGhostArray[j]->flags |= GhostInfo::ScopedEvent;

            // Set up a pointer to the new object.
            NetObject* pObject = 0;

            // If there's a valid ghost object...
            if (mGhostArray[j]->obj)
            {
                // Set up a buffer for the object send.
                U8 iBuffer[4096];
                BitStream mStream(iBuffer, 4096);

                // Pack the server object's update.
                mGhostArray[j]->obj->packUpdate(this, 0xFFFFFFFF, &mStream);

                // Set the stream position back to zero.
                mStream.setPosition(0);

                // Create a new object instance for the client.
                pObject = (NetObject*)ConsoleObject::create(pClient->getNetClassGroup(), NetClassTypeObject, mGhostArray[j]->obj->getClassId(getNetClassGroup()));

                // Set the client object networking flags.
                pObject->mNetFlags = NetObject::IsGhost;
                pObject->mNetIndex = mGhostArray[j]->index;

                // Unpack the client object's update.
                pObject->unpackUpdate(pClient, &mStream);
            }
            else
            {
                // Otherwise, create a new dummy netobject.
                pObject = new NetObject;
            }

            // Execute the appropriate console callback.
            Con::executef(1, "onGhostAlwaysObjectReceived");

            // Set the ghost always object for the client.
            pClient->setGhostAlwaysObject(pObject, mGhostArray[j]->index);
        }
    }
    else
    {
        // Iterate through the scope always objects...
        for (j = mGhostZeroUpdateIndex - 1; j >= 0; j--)
        {
            AssertFatal((mGhostArray[j]->flags & GhostInfo::ScopeAlways) != 0, "NetConnection::activateGhosting:  Non-scope always in the scope always list.")

            // Clear the ghost update mask and flags appropriately.
            mGhostArray[j]->updateMask = 0;
            ghostPushToZero(mGhostArray[j]);
            mGhostArray[j]->flags &= ~GhostInfo::NotYetGhosted;
            mGhostArray[j]->flags |= GhostInfo::ScopedEvent;

            // Post a network event to ghost the scope always object.
            postNetEvent(new GhostAlwaysObjectEvent(mGhostArray[j]->obj, mGhostArray[j]->index));
        }
    }

    // Send the ghosting always done message.
    sendConnectionMessage(GhostAlwaysDone, mGhostingSequence);

#56 Please take the time to make this a resource. Even if you don't support it, it's a great starting point for further development along these lines... you did the hard part for sure. :)

If there were still a HEAD, this is something that should be checked in.

#57 Yeah it shaved off 2mins off my load time. AFX has alot of datablocks.

#58 Daniel, thanks very much!

Your method is wonderful!

#59 Is anyone else having problems with loading a mission, then leaving the mission and loading another mission? Or is that an unrelated problem?

#60 Is someone could do a summary of the update to perform in order to improve the loading time ?

thanks