Vehicle::updatePos - Physics clock too fast

#1 Vehicle physics are processed separately on both client and server. I believe the intent is to produce smooth and relatively lag-free vehicle movement by processing the physics on the client. The server does get the final word, but it will only force an update on the client if it determines that the client has gotten too far from where it should be. Since both client and server are processing the same move events, they should stay mostly in sync without any forced updates from the server.

processTick should be called once per object per tick. However, since Torque maintains separate client and server objects, in any situation where the client and server are the same system you will see two calls to processTick per object per tick. One for the client object, one for the server instance of the same object. These are separate instances of the object and the "duplicate" calls will not interfere with each other.

IIRC, updatePos may be called multiple times per tick, depending on the integration rate. The integration rate comes from the integration data block value. A setting of 1 will call updatePos only once per tick. A setting of 2 will cut the dt value in half and call updatePos twice. And so on. So with an integration value of say 4, running "single player" or as multiplayer host you will see 8 calls to updatePos per tick for every vehicle in the game. 4 for the client instance, 4 for the server instance.

Again, updatePos is intended to run on both client and server objects. Note that the physics time is not doubled in this situation because both the client and server are separately processing the physics for the vehicle.

That is how it "should" work, AFAIK. Does that help at all?

#2 Scott,

Yes. A big help. I did not realize that the client was doing the same physics as the server. I thought instead it was merely interpolating data passed over from the server. I'll go back and take a new look based on this understanding.

Question: With both server and client sides processing the same physics in a single PC game, does this impose an unnecessary computational overhead on the system and, for example, unnecessary reduce the number of vehicles that could be run in a game?

Thanks again,

Kristen.

#3 Answer: For a pure single player game, yes, it is a waste of cpu time. (edit: oops, not so much as I thought. See next post)

Note also that it's more than just the force/drag/mass calculations that are being duplicated. The collision detection functions are also being duplicated. Put it all together and it takes a fair bit of cpu time for each vehicle. I haven't preformed any actual tests so I'm just guessing here, but I would think this could well put a rather cramped limit on the number of vehicles that could be used in game. Especially with integration values set above 1.

This setup is best suited to a networked client/server setup, where the client and server are separate machines. Good for internet-multiplayer games. Not so good for single player games.

Even in a multiplayer environment however, you could save the clients some cpu cycles by performing the collision and physics calculations only on the server. (Thus leaving the client machine more time for rendering.) This would most likely produce some unpleasant "lag" issues, however. For example, a vehicle might appear to pass a foot or two into a wall (on the client) before the next update arrives from the server with the proper collision reaction. How much of a problem that would be, and whether or not it's an acceptable tradeoff would depend on the particulars of each specific game.

#4 Upon further investigation, it seems I was somewhat mistaken. I think the physics are only duplicated on client controlled objects. Network ghosts are simply interpolated. So, no, this isn't a great waste of time, as the client will presumably only be directly controlling one vehicle at a time. Additional vehicles will have their physics calculated on the server only.

I think.

#5 What I just did that made me rethink this:

Add to vehicle.cc @ line 602: (after the "if(mDelta..){")

if (isClientObject()) Con::printf("Interpolating vehicle position");

and @ line 617: (after the "else {")

if (isClientObject()) Con::printf("Processing vehicle physics");

Recompile and launch the starter.racing example. We see that while we're driving the car, "Processing vehicle physics" is printed. But when we switch to an observer camera (Ctrl-c), "Interpolating vehicle position" is printed.

Further testing should be preformed, however this seems to indicate that the server will feed the client updates for vehicles that the client is not controlling, thus preventing the client from running physics calculations on those vehicles.

I only tested this in "single player". I'd like to see what it does over a network connection.

edit:
Hmm. Further tests show that that is not entirely true either. It seems that the client does indeed process the vehicle physics on some of the ticks, even when the vehicle isn't the client controlled object.

#6 Before posting the question, I had spent a couple of days instrumenting gameProcess.cc, Vehicle.cc, and wheeledVehicle.cc with output to clog in order go profile the call sequencing and try to gain some understanding on my own how things worked (before bugging you guys). For what its worth here are some of my observations.

1) gameProcess::advanceServerTime and gameProcess::advanceClientTime did not always compute the same number of ticks (tickCount). The lines computing tickCount are different in each (by design or mistake I don't know). They follow:

bool ProcessList::advanceServerTime(SimTime timeDelta)
{
...
SimTime targetTime = mLastTime + timeDelta;
SimTime targetTick = targetTime & ~TickMask; //*** different ***
SimTime tickCount = (targetTick - mLastTick) >> TickShift;

bool ProcessList::advanceClientTime(SimTime timeDelta)
{
...
SimTime targetTime = mLastTime + timeDelta;
SimTime targetTick = (targetTime + TickMask) & ~TickMask; //*** different ***
SimTime tickCount = (targetTick - mLastTick) >> TickShift;

2) Vehicle::processTick was not necessary called a number of times equal to tickCount but sometimes
by the number of moves associated with the object. This can be seen in ProcessList::advanceObjects.

...
for (m = 0; m < numMoves && pSB->getControllingClient() == con; )
{
obj->processTick(&movePtr[m++]);
}
...

This seem to be associated with whether the camera was slaved to the vehicle or an outside observer (F8).

3) wheeledVehicle::renderImage was called twice in rapid succession for each client display cycle.

There were some other observations. But perhaps this is enough for now. I can set up the diagnostics again and send you the output if that would be of use.

Not sure where this leaves us. I would like to think I could simply introduce an alternative method for computing the rigid bodies position and velocities in Vehicle::updatePos as in the following:

// Integrate forward //!!!TAG: 01-01-2005:
//if (!mRigid.atRest) //!!!TAG: 01-01-2005:
// mRigid.integrate(dt); //!!!TAG: 01-01-2005: //disable physics
//!!!TAG: 01-01-2005: -------------------------------------------------------------------------------
if( mFirstPass )
{
mFirstPass = false ; //all variables beginning with "m" are object members
mInitialPosition = mRigid.linPosition ;
mPhysicsTime = 0.0f ;
}
float lAmplitude = 3.0f ;
float lOmega = 0.1f * 6.28f ;
float lPhase = - 0.5f * 3.14f ;
float lAngle = lOmega * mPhysicsTime + lPhase ;
mPhysicsTime += dt ;
mRigid.linPosition.z = mInitialPosition.z + 3.0f + lAmplitude * sin( lAngle ) ; //simple vertical oscillation

mRigid.linVelocity.x = 0.0f ;
mRigid.linVelocity.y = 0.0f ;
mRigid.linVelocity.z = lAmplitude * cos( lAngle ) * lOmega ;

//!!!TAG: 01-01-2005: -------------------------------------------------------------------------------

// Deal with client and server scripting, sounds, etc.
if (isServerObject()) { ...

However, while the vehicle in general moves up and down as expected, there is considerable "jumpiness" to the image. Is there something else I should be assigning in addition to the rigid bodies position and velocity state?

Hope this has been a good and productive question.

Your legendary responsiveness is well deserved.

Thanks.

Kristen.

#7 I hope you're not expecting too much from me.. Note that I'm not associated with GG in any way, apart from being an SDK owner. I just *thought* I had a good working understanding of these functions. Apparently, you've dug a bit deeper than I.

Regardless, I am rather motivated to figure this out. I am also rewriting much of the vehicular physics for my own game plans.

Anyway, I've been reading through gameProcess.cc and while I won't pretend to understand all of it, from what I can tell what's supposed to be happening is:

processTick is supposed to be called once every 32ms. Each call to processTick gets one Move. Moves are supposed to be 32ms snapshots of user (or AI) input.

Now, ideally, if things are running smoothly, processTick will indeed be called once every 32ms (give or take), with the corresponding Move for the last 32ms. However, this isn't always possible. If the computer is especially busy, say for example re-lighting the mission in the mission editor, it may miss one or more ticks. If the network connection is laggy, moves may arrive late. In either case, the advanceClientTime and advanceServerTime functions will run the appropriate number of ticks to catch up.

This is desirable behavior, IMO, because it doesn't allow network lag or a busy cpu to "remove" time from the simulation. As soon as the problem clears, the game is advanced to catch up, with no loss of physics steps or input events.

That is how I believe it is supposed to work. I think that may explain some of the behavior you are observing. Although I can't be sure it's working entirely correctly, but I also haven't found anything wrong with it yet.

Will keep looking...

#8 I don't have anything to contribute to your conversation here (you guys are already way above my understanding of the processing involved), but I did want to say that it is awesome that someone is looking at stock TGE's physics handling with a light towards making it even more accurate--it's a model right now that worked very well for Tribes, and works very well in general for FPS games, but as it appears from your discussion, it could use some reality checks, and having someone really analyze it from a fresh viewpoint is quite a boon to TGE!

Keep it up--while the convo may be just between you two at the moment, I'm sure that many are listening closely!

#9 I agree with Stephen, this is very interesting stuff. I've been looking to upgrade the physics a bit, and this thread gave me some good ideas of where to start.

#10 Scott, you have contributed some useful observations. I am going to go back and reinstrument/analyze the situation with these new understandings and will post what I find out.

Some background on these efforts. I worked for Mechanical Dynamics Inc. for most of the eighties doing constrained rigid-body dynamics modeling of all sorts of mechanisms (e.g., circuit brakers, compressors, door mechanisms, etc.) including large full vehicle models with detailed tire models. However, their modeling tool was for engineering analysis, was slow and non-interactive. After leaving MDI, I developed a dynamics library, Tango, that is on the other hand very fast, interactive, with a broad modeling interface and have been licensing it in the CAD/CAE (that is engineering) markets. All the demos use low-tech DirectX graphics and I was looking for a platform to show-off Tango's capabilities when I came across the TGE.

Given the cursory interest expressed here, I am beginning to wonder if Tango could not be of benefit to Torque developers as well. I believe strongly in collaborative working relationships that bring together diverse talents. I would be willing (enthusiastic) to provide Tango free to a limited number of developers who have both the motivation and the capabilities on the game development side to help determine what is needed, how it is packaged and priced, and then use the resulting tool(s) successfully in their own games (again free of charge or royalty). You mentioned vehicle physics. If you are interested, I have a DirectX demo of a Tango LOD vehicle that simply follows the terrain when distant, changes to a single free body physics model with contact/tracking wheels at mid distances, and again changes to a constrained, multi-body physics vehicle in the foreground. The additional bodies are separate wheel suspension assemblies that move with respect to the vehicle body. As well, the changes are smooth and on the fly. I can zip and send to you if you wish. (other capabable game developers listening, that go's for you too)

Regardless, again, thanks for your continuing input. I'll keep posting what I turn up.

Kristen.

#11 I would be interested in this as well, could try and assist in figuring out the complete issue as well...
been thinking on fine tuning the vehicle physics and the whole vehicle update issues, as you have found there has been past issues with using vehicles as more than what Tribes used.

#12 @Kristen: You should most definitely contact Josh Williams regarding your ideas and experience. He's in charge of third party development for GarageGames, and I'm 100% positive he'd love to know about your experience and interests!

Various types of physics have been very hot topics in the past, and come up quite often, so there really is a big need for the experience and motivation you have!

Edit: Link fixed, and thanks! That's what I get for posting during a conference call ;)

#13 Yes physics has been a hot topic lately.

Oh a fix for Stephen's post.


You should most definitely contact :
Josh Williams

#14 Regarding the difference in the "targetTick = ..." lines in advanceServerTime and advanceClientTime (gameProcess.cc): The difference appears to have no real impact as far as I can tell. Here are some actual numbers taken from those functions:

advanceServerTime():

timeDelta    targetTime    targetTick    tickCount

  1024          1024          1024          32
    18          1042          1024           0
     3          1045          1024           0
     3          1048          1024           0
     3          1051          1024           0
     3          1054          1024           0
     3          1057          1056           1
     3          1060          1056           0

advanceClientTime():

timeDelta    targetTime    targetTick    tickCount

  1024          1024          1024          32
    18          1042          1056           1
     3          1045          1056           0
     3          1048          1056           0
     3          1051          1056           0
     3          1054          1056           0
     3          1057          1088           1
     3          1060          1088           0

These are actual values for timeDelta, targetTime, targetTick, and tickCount printed to a log each time the functions were called. Note that the initial values are larger. This data was taken from the very beginning of my log file, when the engine was first starting. I left the first few values in there as an example of what happens when the engine is late by multiple ticks, and needs to "catch up".

Note the only difference is that the client is one tick ahead of the server. While the server advances the tick to catch up with the time, the client advances the tick to stay ahead of the time.

Now, I haven't found anywhere where this would make any difference. I can't say for certain that no one is counting their total ticks. I suppose somewhere the server may say to the client "I'm on my 47th tick, where are you?", but I haven't found any such code. So far it seems to me that the difference really makes no difference. :-P

Either way, both client and server seem to be keeping accurate time, and advancing by 1 tick per 32 milliseconds, as they should.

So... Moving on:

The function advance...Time() will call advanceObjects() [tickCount] times each time they're called. Rephrased that says advanceObjects() will be called once per tick. Now it is the job of advanceObjects() to decide how and when to "tick" each object (which it will do by calling obj->processTick(), or not, for each object).

AdvanceObjects() checks two things for each object. First it checks to see if it is of type ShapeBaseObjectType (line 217), then it checks to see if a client is controlling the object (lines 220 and 222).*

IF and only if both conditions are true, then the objects gets a call to processTick() for each Move event pending for the controlling client. (Note that Move events should also be collected at the rate of 1 per 32ms, I think.)

ELSE the object simply gets a single call to processTick(), with no associated Move.

Either way, each object should get one processTick() per 32ms. Either directly from the above mentioned tick counters, or as Move events come in.


* IMO this seems a rather convoluted way to perform this check, since it basically says "Does this object have a controlling client, AND is that client controlling this object?". That strikes me as a little redundant. Can there really be any case where an object has a controlling client, but the client isn't controlling the object? That seems rather contradictory. Wouldn't that only happen if the engine screwed up?

#15 Oops, missed a piece of the puzzle...

AdvanceObjects() is only called once per 32 milliseconds (well, once for the client, once for the server). But (I think) in order to process Moves as quickly as possible, advanceClientTime() also has the ability to directly tick objects. AdvanceClientTime may do one of two things (based on the IF on line 138):

IF it's time for a tick, then advanceObjects() will be called.

ELSE if the client is controlling an object, then that object may be ticked right here and now for each Move pending.

Note that this should still maintain the 1 tick per 32ms rule, since Moves should be 32ms blocks. Regardless of whether they are used in advanceClientTime() or in advanceObjects(), once used they are cleared from the Moves list. Each Move is therefor only used once.

(Something's still bothering me here.. More on this soon.)

#16 Scott,
Your last two submissions look like a lot of good detective work. I'm going to go through them in detail along side the code. One statement did catch my eye "Either way, both client and server seem to be keeping accurate time, and advancing by 1 tick per 32 milliseconds, as they should." I've noticed that, while the vehicle is being controlled, the server physics is often advanced by the number of moves which is often not the same as ticks. Please see the following trace that picks up at processList::advanceServerTime and advanceClientTime. Note the first sequence, two server ticks and only four (one tick's worth) physics evaluations (Vehicle::updatePos). Is this really as it should be? I've also seen the inverse, one tick and eight physics evaluations. While it may all work out in the end, it makes it difficult to understand how to interface another physics system to the TGE. (more later as well)

ProcessList::advanceServerTime( 47 )
| mLastTime = 15025 targetTime = 15072 targetTick = 15072 tickCount = 2
| ProcessList::advanceObjects
| | (obj->mTypeMask & ShapeBaseObjectType)
| | | (con && con->getControlObject() == pSB)
| | | | numMoves = 1
| | | | Vehicle::processTick
| | | | | (mDelta.warpCount >= mDelta.warpTicks)
| | | | | | Move input
| | | | | | Vehicle::updatePos
| | | | | | Vehicle::updatePos
| | | | | | Vehicle::updatePos
| | | | | | Vehicle::updatePos
| | | | | | mDelta.pos -160.839 913.297 195.376
| | | | | | mDelta.posVec 0.0114441 -0.268494 0.00799561
| ProcessList::advanceObjects
| | (obj->mTypeMask & ShapeBaseObjectType)
| | | (con && con->getControlObject() == pSB)
| | | | numMoves = 0
ProcessList::advanceClientTime( 47 )
| mLastTime = 15025 targetTime = 15072 targetTick = 15072 tickCount = 1
| (connection)
| Vehicle::interpolateTick( dt = 0)
| ProcessList::advanceObjects
| | (obj->mTypeMask & ShapeBaseObjectType)
| | | (con && con->getControlObject() == pSB)
| | | | numMoves = 1
| | | | Vehicle::processTick
| | | | | {mDelta.warpCount >= mDelta.warpTicks}
| | | | | | Move input
| | | | | | Vehicle::updatePos
| | | | | | Vehicle::updatePos
| | | | | | Vehicle::updatePos
| | | | | | Vehicle::updatePos
| | | | | | mDelta.pos -160.851 913.577 195.368
| | | | | | mDelta.posVec 0.0119476 -0.279968 0.00814819
| Vehicle::interpolateTick( dt = 0)
| Vehicle::advanceTime( dt = 0.047)
Vehicle::interpolateTick( dt = 0)
Vehicle::interpolateTick( dt = 0)

#17 Quote: One statement did catch my eye "Either way, both client and server seem to be keeping accurate time..."

Caught me making assumptions. :-o

I read somewhere that Moves are supposed to be input captures in 32 millisecond intervals. I've been proceeding on the assumption that it was true. BUT I haven't confirmed it yet. My bad.

#18 @Kristen:

You're definitely digging very deep here. I'm delighted to see someone evaluating the vehicle code this closely - with so many other technical fires going on (like TSE, TNL, 1.4 update, doc work, etc.) it's well-nigh impossible for us to budget the time to take this close of a look. So... Great job! Thank you! :)

I want to take a look at this thread more closely, as I think I (or other GG employees that I can show this thread to) can probably shed a bit of light on the nature of Torque's time-processing and help you and the rest of the community resolve this issue.

Thanks,
Ben

#19 @Ben:

Thanks for the attention. I look forward to your continued input as this is certainly not a trivial piece of code to understand. I have added more detail to the trace that should be of benefit. Also, after spending some time reviewing the trace, I believe I can put forward some concrete and hopefully helpfull observations.

At the end of this message, I provide a program trace for the "starter.racing"
example supplied with the Torque engine. Each level of the trace is preceded with
a statement sequence header that is either a class method or conditional
statement. Variable values of interest are also streamed out. The trace has multiple
start points as documented in the left most level headers.

While many observations can be made, I provide three below that I believe represent errors
or at the very least highlight unintentional behaviors and performance losses. I am
interested in determining whether or not these observations are correct, what they
really mean and if any corrective actions are in order.

1) In ProcessList::advanceServerTime, the physics is not always advanced the same
number of ticks as the tickCount. I think this is an error. ProcessList::advanceObjects is indeed called
once for each tick as expected. However, inside ProcessList::advanceObjects,
Vehicle::processTick is called based upon the number of moves. Consequently,
Vehicle::processTick is called multiple times per call to ProcessList::advanceObjects
and sometimes not at all. The net effect is the physics is sometimes advanced
more ticks than tickCount and sometimes less. I think this is an error.

2) Vehicle::interpolateTick gets called four times per render cycle, twice from
ProcessList::advanceServerTime and then twice from GameConnection::writePacket.
I believe three of the calls are unnecessary.The results (mDelta values) are
identical in the two calls from GameConnection::writePacket
and the second call from ProcessList::advanceServerTime. It appears that the first
call in ProcessList::advanceServerTime and both calls from GameConnection::writePacket
have no effect on the vehicle position. If not an error, I believe this behavior is
at least unintentional and consumes unnecessary cpu cycles.

3) [b]WheeledVehicle::renderImage gets called twice per render cycle and with the
same vehicle position state. The second call follows qhickly within 10 milliseconds.
If there is nothing to be gained by two calls, I believe this is an unnecessary
expenditure of cpu cycles.

Please see the following post for the trace.

#20 ProcessList::advanceServerTime( 46 )
| mLastTime = 17495 mLastTick = 17472 targetTime = 17541 targetTick = 17536 tickCount = 2
| ProcessList::advanceObjects
| | (obj->mTypeMask & ShapeBaseObjectType)
| | | (con && con->getControlObject() == pSB)
| | | | GameConnection::getMoveList: movePtr 01377300 numMoves 1 this 0012F380
| | | | Vehicle::processTick
| | | | | (mDelta.warpCount >= mDelta.warpTicks)
| | | | | | Move type 'input'
| | | | | | Vehicle::updateMove
| | | | | | | throttle, steer x & y: 1, 0.319835, -0.139784
| | | | | | Vehicle::updatePos (called 4 times here)
| | | | | | mDelta.pos -152.326 923.856 195.071
| | | | | | mDelta.posVec 0.183731 -0.478638 -0.00177002
| | | | GameConnection::clearMoves count = 1 this = 0012F380
| ProcessList::advanceObjects
| | (obj->mTypeMask & ShapeBaseObjectType)
| | | (con && con->getControlObject() == pSB)
| | | | GameConnection::getMoveList: movePtr 01377300 numMoves 0 this 0012F380
| | | | GameConnection::clearMoves count = 0 this = 0012F380
ProcessList::advanceClientTime( 46 )
| mLastTime = 17495 mLastTick = 17504 targetTime = 17541 targetTick = 17568 tickCount = 2
| (connection)
| Vehicle::interpolateTick( dt = 0)
| | mDelta.pos -152.326, 923.856, 195.071
| | mDelta.posVec 0.183731, -0.478638, -0.00177002
| | Vehicle::setRenderPosition( pos ): pos.x, y, z -152.326, 923.856, 195.071
| GameConnection::collectMove time = 17504 this = 01373720
| | pushMove
| ProcessList::advanceObjects
| | (obj->mTypeMask & ShapeBaseObjectType)
| | | (con && con->getControlObject() == pSB)
| | | | GameConnection::getMoveList: movePtr 01376B30 numMoves 1 this 0012F2A4
| | | | Vehicle::processTick
| | | | | (mDelta.warpCount >= mDelta.warpTicks)
| | | | | | Move type 'input'
| | | | | | Vehicle::updateMove
| | | | | | | throttle, steer x & y: 1, 0.449744, -0.109871
| | | | | | Vehicle::updatePos (called 4 times here)
| | | | | | mDelta.pos -152.508 924.343 195.073
| | | | | | mDelta.posVec 0.181702 -0.486938 -0.00195313
| | | | GameConnection::clearMoves count = 1 this = 0012F2A4
| GameConnection::collectMove time = 17536 this = 01373720
| ProcessList::advanceObjects
| | (obj->mTypeMask & ShapeBaseObjectType)
| | | (con && con->getControlObject() == pSB)
| | | | GameConnection::getMoveList: movePtr 01376B70 numMoves 1 this 0012F2A4
| | | | Vehicle::processTick
| | | | | (mDelta.warpCount >= mDelta.warpTicks)
| | | | | | Move type 'input'
| | | | | | Vehicle::updateMove
| | | | | | | throttle, steer x & y: 1, 0.449744, -0.109871
| | | | | | Vehicle::updatePos (called 4 times here)
| | | | | | mDelta.pos -152.687 924.838 195.075
| | | | | | mDelta.posVec 0.17897 -0.4953 -0.00210571
| | | | GameConnection::clearMoves count = 1 this = 0012F2A4
| Vehicle::interpolateTick( dt = 0.84375)
| | mDelta.pos -152.687, 924.838, 195.075
| | mDelta.posVec 0.17897, -0.4953, -0.00210571
| | Vehicle::setRenderPosition( pos ): pos.x, y, z -152.536, 924.42, 195.073
| Vehicle::advanceTime( dt = 0.046)
GameConnection::writePacket
| Vehicle::interpolateTick( dt = 0)
| | mDelta.pos -152.687, 924.838, 195.075
| | mDelta.posVec 0.17897, -0.4953, -0.00210571
| | Vehicle::setRenderPosition( pos ): pos.x, y, z -152.687, 924.838, 195.075
| Vehicle::interpolateTick( dt = 0.84375)
| | mDelta.pos -152.687, 924.838, 195.075
| | mDelta.posVec 0.17897, -0.4953, -0.00210571
| | Vehicle::setRenderPosition( pos ): pos.x, y, z -152.536, 924.42, 195.073
WheeledVehicle::renderImage
WheeledVehicle::renderImage