Vehicle::updatePos - Physics clock too fast

#21 If I'm reading this correctly, this will not just effect the vehicles, but should be a major speed/frame rate improvement for the whole app, as long as any objects are on screen and are being ticked by the engine. 5 or so redundant calls for each object? Is this possible?

#22 RE: 2) Vehicle::interpolateTick gets called four times per render cycle, twice from
ProcessList::advanceServerTime and then twice from GameConnection::writePacket.

Looking at the source, I don't see any calls to interpolateTick from within advanceServerTime. I do see two calls from advanceClientTime, however. The first is called only if it's time for a tick, or if moves are pending. The second seems to be called no matter what. Looks like the first call is rather pointless, since before anything is rendered, it will be called yet again. The only reason I can think of to call interpolateTick before processTick would be if processTick uses one or more values set in interpolateTick. Which, IMO they should not do, but some classes may.?

As for calling interpolateTick from writePacket... writePacket says:

mControlObject->interpolateTick(0);
sum = mControlObject->getPacketDataChecksum(this);
mControlObject->interpolateTick(gClientProcessList.getLastInterpDelta());

Here we're interpolating to 0 (which I think is the predicted interpolation of at the next future tick), then calling getPacketDataChecksum, then interpolating back to the last "correct" interpolation position. Why? Well...

For all ShapeBase objects, getPacketDataChecksum then calls writePacketData, which grabs various values from the object such as position, velocity, energy, and such. Now, idealy (IMO) writePacketData should only be grabing values that are changed on a per-tick basis. In which case there's no point to calling interpolateTick. However, if any ShapeBase object (or object of a class derived from ShapeBase) is using values that are changed by interpolateTick, then the calls to interpolateTick may serve an important purpose.

Frankly, I don't think it's really appropriate to be calling interpolateTick from a writePacket function. But just removing those calls *may* cause unexpected behavior from some objects. If that's the case, I think those (derived) ShapeBase objects should be "fixed", and the interpolation calls removed from writePacket.

Edit: I'm going to go ahead and remove all those duplicate calls to interpolateTick, leaving only the second one in advanceClientTime. I believe that will work for me, however I'm no longer using any of the default derived ShapeBase classes, so if one of them breaks I may not know... I think the change should work fine with my own Ship (rewrite of Vehicle) class. Will post results later..

#23 Hi Kristen. It's great your digging into the code so deeply. I'm sorry that I don't have time to reply right now, but I wanted to pop in and say good work so far. I know Ben and I would both like to reply here as soon as we have the time, and I'm sure Pat, Rob, or Tim Gift would as well. I'll ask Tim if he can stop by specifically.

Hopefully one of us, or someone like Clark who's very familiar with this stuff, will be able to get a detailed reply to you soon.

We'll be interested to see what you come up with as you continue on in your investigations. We have a lot of exciting things planned for vehicles and for GameBase-dervied objects in general for the next version of Torque. It looks like you really know what you're doing, and are learning the engine's handling of networked physics and simulation rapidly. If you want to get in touch via email, I'd be happy to talk with you about some of this stuff in more detail.

#24 More detail here instead of email, please!

I'm sure I'm not the only one who is also digging into this same code for different reasons, but finding extremely helpful insights at the same time. I just found this thread today, but the process of discovery has been almost identical to things I've been pulling apart. At this point, I'm not sure if my own AI-related problems are more or less related to connection object handling or physics simulation, but if my investigations point me this way I'll definitely be contributing!

#25 Russ: I'll continue posting whatever I find out here as you request. This "Clark" guy, that's not Clark Kent is it?!

Josh: Thanks for the "pop-in". Think I'll take you up on your email on some related issues. But not just yet. Was out today and am still catching up. This "Clark" guy, that's not Clark Kent is it?!


Scott: Thanks. Some very useful observations. I'm going back in with this new info.

I did observer a couple other things in my copy of the trace that contained the above piece as well as many more cycles. The final value of vehicle position calculated in Vehicle::updatePos for the server object was always the same (for the many cycles I looked at) as the previos position computed for the client object. I further noticed that the number of messages for the server object (which was controlling the number of calls to Vehicle::processTick) was the same as the number of moves (and ticks) for the prior client object pass. Consequently, the server object was always making the same number of passes (regardless of its own tickCount) on Vehicle::processTick as the prior client object pass and therefore just keeping up. Actually, Scott makes a similar statement earlier on in this thread. Finally, I noticed the client object actually receives the first non-zero value of dt, not the server object, which seems consistent with the server just keeping up.

IMO, what's a little wierd about all this is that the code seems to work fine but not as it appears that it should; that is given the variable names, method names, etc.

Hope this is of use to some folks out there. Gotta go take a closer look at how messages get from the client (which collects them) to the server side of things. Later.

#26 Havent got too much time to post here. But just FYI, the reasoning for the interpolate tick method is to interpolate from the last calculated delta position to the next.

The weird thing is that it calculates the delta values "backwards", so that it steps from A to B where B is the "final" position and A is the current position.

So each tick, you get the delta between current values and new values, calculate the delta, then send it to the client. Once on the client, it calculates those delta's (hmm, that seems unlikely, I might be off base there, it might not recalc the delta). So then it takes the delta value and interpolates forward from new position - delta to newposition.

Now that would lie well with what you observed (where the server position is static while the client interpolates often, then they should both change). The concept is that both client AND server are ticked, then ONLY the client is ticked for interpolation.

Dont forget the integration rate might also mean that the vehicle is ticked more times then a game tick (so you should maybe instrument the physics tick rate outside of the vehicle process calls).

#27 I'm not sure I've written this very clearly, but here's my take on it...

The engine advances time forward at a fixed 1/32 rate through calls to processTick. Control objects receive a single move for each tick, and basically see time as a stream of moves. Each call to advance time will tick forward the requisite number of ticks needed to cover the real time interval from the last advance time call (rounding up). Since real time rarely advances at a 1/32 interval, objects are interpolated backwards from their latest tick position to the actual "real" time and rendering is normally done from one of these interpolated inter-tick positions. This means objects are always slightly ahead of real time and what is being rendered. Originally objects didn't maintain a completely seperate states for their "interpolated pos" vs. their current ticked position, which is why there are calls to interpolateTick(0). The interpolate 0 essentially resets an object back to it's current tick position in order to either tick forward, or extract information (as in the network packing functions). The lack of seperate states was a bad design decision which was never fully corrected.

So, the basic flow of time is a series of processTick calls, interspersed with calls to interpolateTick. Since control objects always receive a single move per tick, they see time as a series of moves, which is how control objects are synchronized with the server. Every move produced by the client is processed both by the client object, and it's server clone. Both client and server objects are started from the same state, and fed the same moves, which normaly keeps them in sync (assuming your move processing and physics are determanistic). They don't process moves the same way though. The client normally processes each moves as it's generated, but the server object spends most of it's time waiting for packets from the client. The server control object is not ticked forward during this wait and is basically frozen waiting for a client update. When moves from a client are received, they are processed in bulk by the server in order to quickly bring the control object up to date. Basically each control object on the server is running in it's own time line.

This next part is what causes most of the client's advance time complications... Syncing between the client and server involves syncing two things, the state of the control object, and it's position in time. Since control objects see time as a series of moves, this later step involves syncing moves. When the server detects a synchronization problem, it updates the client with the control object's state and the last move time processed. When the client receives this update, it first sets the object state to that received from the server... This new state will effectively be back in time as far as the client is concerned, but since it also receives the move time that generated that state, it can advance the object forward by replaying all client moves generated since that time. The client maintains a history of moves for this purpose. So, normally a client control object advances one processTick every 1/32 of client time, but when a server update is received the object is reset to the server's state and time, then fast-forwarded to the current client time.

#28 Notes...

1. Client time is advanced at 1/32 fixed rate, and interpolation is used to render inter-tick frames.

2. Control objects always receive one move per tick and see time as a stream of moves.

3. Moves are transmitted to the server and played on the control objects as they are received. Non-control objects on the server receive a tick every 1/32 of a second, but control objects only advance their time (in spurts) when they receive moves from their controlling client.

5. On the client, all objects normally receive one tick every 1/32 of a second (interspersed with interpolateTick and render calls), including the control objects which receive their one move per tick.

6. Client and server control objects are normally in sync since they each start from the same known state, and each processes the same stream of input moves representing the same elapse time.

7. When the server updates a client's control object, it sends the client it's current state and which move was last processed. When the client receives this update, it sets it's control object to the received state, and then processes (in-bulk) all the client moves generated since the move associated with the server update.

Some notes on the physics...

1. This state rewind/fast forwarding, along with the asynchronous advancement of control object time on the server plays havoc with physical interaction between dynamic objects.

2. Ghost object vehicles perform perform both interpolation, and physics based prediction... when a ghost receives a position from the server, it interpolates from it's current position to this new position. During this time, no physics calculations take place, though the wheels will be extended to touch the ground. Once the vehicle reaches it's new position, if no further server updates are forthcoming, then the ghost will simulate forward (using the normal physics processing) using the last client move received from the server.

3. Wheel extension is not part of the physics simulation. This was skipped in order to reduce the size of the server sync updates (four less floats to write out). The wheels are always extended full out every tick.

On the double calls to render... during the scene graph building phase, visible objects register to be called for rendering, and they can register more than once. If an object contains translucent materials, it will register itself twice; once so that it can render it's opaque surfaces and once to render it's translucent surfaces.

#29 Phil and Tim: Thanks for the copious notes and information. They are of great help. Tim: regarding your statements about the state rewind & fast forwarding and asyncronous control object time creating problems for dynamic object interaction; my sense of the code is that it was originally designed for discrete event (e.g., orc changes velocity) rather than continous processing as one sees in real-time, interactive dynamics. I've worked with the latter approach for years and so am quite familiar with it but am new to the discrete/asyncronous approach. I'll continue to study it in light of the new information you have provided.

Any plans to re-visit the TGE's architecture to accomodate and better coordinate both approaches?

#30 This approach was specifically designed to meet our multiplayer networking requirements, physics simulation was definitely second :) The main feature of this approach is that every move made by a client control object is simulated, vetted and corrected by the server. No mean feat considering how smooth and responsive the control objects can be, even under lagging network traffic. But if this is a feature you don't need, then you probably don't want to be saddled with the limitations. If the application you are working on is single player only, I'd recommend skipping the whole move client/server syncing process and implementing a new advance time function. I'm pretty sure this is the approach taken by the "Tube Twist" and "Rocket Bowl" teams, both of which have integrated the ODE physics engine into Torque.

At some point there will be an official "single player only" simulation loop, but it's not on anybodies immediate task list.

#31 Any hints as to where those of us that are making single player games can look to optimize this code for our projects?

#32 This is hardly worth mentioning, but...

In advanceObjects() lines 217-235:

if (obj->mTypeMask & ShapeBaseObjectType) {

	 ShapeBase* pSB = static_cast<ShapeBase*>(obj);
	 GameConnection* [b]con = pSB->getControllingClient()[/b];

	 if (con && con->getControlObject() == pSB) {
		Move* movePtr;
		U32 m, numMoves;

		con->getMoveList(&movePtr, &numMoves);

		for (m = 0; m < numMoves && [b]pSB->getControllingClient() == con[/b]; )
		   obj->processTick(&movePtr[m++]);

		con->clearMoves(m);

		continue;
	 }
  }

set A = B, then if B == A?

As far as I can tell, neither con nor pSB are changed between these lines. So that conditional in the for block should always be true. Isn't that rather pointless?

#33 RE: "Originally objects didn't maintain a completely seperate states for their "interpolated pos" vs. their current ticked position, which is why there are calls to interpolateTick(0)."

Thanks Tim, I was wondering about those.

I did finnaly remove all the calls to interpolateTick() except the last one in advanceClientTime() on line 184.

Removing the two calls in gameConnection.cc seemed to cause no issues. Removing the first call in advanceClientTime() on line 135 however, made for very un-smooth movement of my control object. Removing the block of code from lines 166 to 177 (including the other processTick() call) seems to have fixed that. Allowing me to remove the first interpolateTick() call. Now moves are only processed on the tick. I don't think this will cause any problems however, since all moves in the list will still be processed, it's just a matter of when.

So far, so good. I tested the changes in a single player game, and a lan game with 1 other client. No issues that I could see. Not exactly a good stress test, but it's the best I can do at the moment.

#34 Re: In advanceObjects() lines 217-235: GetControllingClient will not necessarily return you the control object. I think mounted vehicles (and other remote controlled objects) can have a controlling client set, but are not actually the control object (which is still the player). Can't remember exactly.

#35 That would explain the line 222 "if (con && con->getControlObject() == pSB)", because that translates to "If this object has a controlling client, and that client's control object is this object", correct? But I'm looking at line 228 "for (m = 0; m < numMoves && pSB->getControllingClient() == con; )".

This is what doesn't make sense:

line 220: con = pSB->getControllingClient()

line 228: pSB->getControllingClient() == con

If neither pSB nor con are changed between those lines, there's no point in the conditional on line 228 because it will always be true. No?

EDIT: Unless the ShapeBase object in question manages to change it's mControllingClient between advanceObjects() lines 220 and 228. But that would only be possible if Torque is multithreaded, yes? I haven't seen anything to suggest that it is multithreaded... ?

#36 @Tim, great summary. That should be made into a resource.

@Scott, in regards to the "set A=B, then if B==A" code...it is possible that process tick executes a script which changes the control object on the connection, so it isn't actually tautological. This prevents you from doing something like, switching control objects and then pushing some key and having that key apply to the old control object. Although in that case it still looks like a bug since the moves would instead be lost. BTW, notice that this code means an object cannot delete itself during a process tick! I'm sure many reading this thread have encountered problems with that in the past (forcing them to schedule an object deletion rather than doing it directly).

#37 Ah yes. Of course, because it's in the for loop. Duh. :-P That's what I was missing, thanks.

#38 Just a followup to my post regarding Moves being captured in 32 millisecond intervals... I know Tim already confirmed this, but for whatever it's worth to anyone following this, gameProcess.cc line 161 is where you'll find the call to collectMove(). This is in advanceClientTime, in a for("each tick") block (line 140).

This doesn't necessarily guarantee that Moves are collected once every 32 milliseconds, but it does guarantee that Moves are collected once per client tick.

#39 RE: "No mean feat considering how smooth and responsive the control objects can be, even under lagging network traffic. But if this is a feature you don't need, then you probably don't want to be saddled with the limitations."

Tim, I've seen more than one glowing review on Torque's network capabilities and actually would like to not exclude that possible context. Consequently, I am endeavoring to work as much as possible with the existing architecture and learn how to play to its strengths. I've just been at this off and on over the last couple of weeks but I am making progress with my objectives, thanks in part to the generous flow of information in this thread. Kudos to the community.

#40 One area that could probably benefit from a good look is the actual physics updating code, which seems to have had some subtle bugs in it for some time now. Just to point out that there's probably some good wins to be had deeper in there. :)