Interpolation problems? (Jumpy players)

#21 I'm glad this thread has been started because I've been wondering if there should be adjustments in the engine code for faster-paced games. AND BTW, I think I have seen reports of the bounding box issue for regular players as well elsewhere on the forum.

I belive that Torque does 3 things to compensate for latency:

prediction
interpolation
and ???

There should be a doc or part of a doc somewhere that describes torque's networking model, but I cant seem to find it online (not at home). But I do suspect that there are some settings that should be tweaked based on the "speed scale" of an individual game...

-s

edit: found the info:

* Interpolation is used to smoothly move an object from where the client thinks it is to where the server says it is.

* Extrapolation is used to guess where the object is going based on its state and rules of movement.

* Prediction is used to form an educated guess about where an object is going based on rules of movement and client input.

http://www.garagegames.com/docs/torque.sdk/engine/Networking.php

#22 We (Digital Dave and I) are using HEAD from 1/04/2004. I update fairly regularly to pick up the latest changes. I have made a number of changes to the engine, both to the wheeled vehicle class and elsewhere.

We are using a slightly modified aiclient.cc to control the AI vehicles.

Our datablocks are pretty standard, although the maxWheelSpeed is set quite high (70).

#23 Stephen there is a link to information on The Tribes networking model in my latest .plan file.

#24 Thanks for the additional info.

Just want to clarify a bit more. It is stated above that when speed is turned down, these effects aren't noticeable. I'm basically trying to figure out if there is a discontinuous point at which this behavior comes into play, or if it increases along with speed.

Of course, wacky physics and collisions can happen at high speeds, in any engine I know of. As the speed is turned up, do things get weirder? As it turns down, does the weirdness subside smoothly? Etc. Any other significant things you notice about the behavior will be helpful. Use your debugging skizzles. :)

I will try to keep looking at this and figure out what's going on. (Again, I'm busy with lots of other stuff, and I'm not yet real familiar with these parts of the codebase, so I can't promise anything). Please continue to post any relevant info.

It may just be that the games are set up so that things are moving too quickly. In that case, maybe the solution is as simple as translating units. Or maybe it'd be viable to derive a SpeedyVehicle class or something and do some special synch coding. There could be many solutions.. just need more info to figure it out.

Sorry that I can't devote more attention to the problem at hand, I know it's frustrating. But it'll be nice once it all gets figured out. I hope some others out there can help out too, since it'll get solved much faster that way.

#25 Awesome, thanks for all the info Dave.

It sucks that it is taking so many days to resolve this. It would've taken about 10 minutes to get to this point if we were in a room working the problem together. :) But such is life on forums.

Given the information above, I'd imagine that this would take about an hour to solve. But since I probably have about 15 minutes a day to spend here, that means it could take 4 days (if you guys don't resolve it yourselves and noone else steps in to help). Sorry if that sounds flakey, but it's the honest truth. For example, right now it's 2:40am, and I have to be at work for my day-job in 5 hours. Ack.

Anyway, I can't devote time to debugging this issue tonight, but I can maybe help point you guys in a useful direction to work it more:

Representing fast-moving objects in simulations or games is a common challenge. However, the solution to this problem is probably pretty simple to come up with. Here's the first thing I'd test:

Is the behavior invariant under unit transformation?

In English, what I'm asking is whether the wacky behavior is dependent upon the size of the measuring unit being used. Explictly, if you double the size of the world units, (so that the actual distance traveled is the same, but the numerical representation of speed is halved) does the bad behavior go away? I don't know at the moment exactly how Torque handles the relevant calculations, so this may be a factor.

To determine whether this is the case, you could literally process an existing level to test it out, or set up a simple dummy test mission, or you could dig through the code and analyze the calculation math.

If the behavior is not dependent on unit size, then it's likely that something slightly more fundamental is going on. This would be strange; I know Mark has said that Torque vehicle physics and networking have been tested in-house a lot, and worked, and obviously they work well in other projects (though they have problems in others, as we've seen in this thread).

Still, there are many potential solutions if this is the scenario. Implementing these solutions would be slightly more work than the nice little numeric dependency, though. :)

If you guys are not comfortable performing these kinds of tests, or digging through the code, I'm sorry I don't have time to help out more directly right now. Maybe I can this weekend, if it won't kill you to wait that long. Or hopefully somebody who's already familiar with this problem will jump in and give you a solution.

#26 It would be nice to know which stage this problem is coming from. If it doesnt happen on local clients, would that mean that this effect is coming from the Prediction stage? And where in the code is this happening? Is it "interpolateTick" or something else??? I'm not very well versed in this part of torque, but it would be nice to know where it is coming from.

Also, could someone post their datablock and dts files for this for others to test if they feel so inclined?


-s

#27 I would just like to add that I get the exact same problem when my AIPlayers run forward. So I suspect that this is a pretty deep problem, not something specific to vehicles. Can anyone else confirm this?

I would like to note that the SERVER bounding box seems to be ahead of the player when he runs forward (significantly ahead, the center of the bounding box is moved almost a meter forward) However, the CLIENT bounding box seems to be in exactly the correct place. (I am simply drawing these two boxes to the screen when playing the game on a server I've hosted on my machine and connected to with the same machine, ie, the normal way of starting torque, a non-dedicated server)

#28 Hey guys, just checking in.

We've talked about this thread in the office a bit.

If anyone is weirded out by the fact that the boxes don't match the displayed graphics, don't worry. That is not the cause of the problem, and is how things are supposed to work. The bounding box is server-side and of course client and server won't always match exactly.

The hopping/warping thing is strange though. In order to take a look at this, here's what we need:

Somebody who is having these problems should download a CLEAN version of head, and reproduce this behavior. We really can't debug anything if there were custom C++ changes to any of the relevant parts of the code. So, if someone can reproduce the behavior on a clean build, we can attempt to investigate it. Send us the journal and we can work it. Otherwise, there's no point because we're working from different bases. Nobody here has observed this behavior on a clean build, so we need this reproduction.

Thanks. If it turns out that you can't reproduce the behavior on a clean build (which I'm not saying will necessarily be the case), then that's a good thing to know-- you will know to figure out what is going on with the changes, or how they interact with the rest of the engine code, to cause this strange behavior.

Thanks, keep us posted here on whether you are able to get this going or not.

#29 If this is a lan only problem, I suppose increasing packet sizes shouldn't be too much of an issue. However, if its in all network type play, then by increasing packet size too much, you could limit 56k users from net play... which isnt a good thing.

#30 Thanks Dave. :)

Yes, I agree it'd be strange if several different teams all independently produced the same strange behavior due to code changes. But it's not impossible, of course. Say, if some resource was used by everyone, or even if a few people just came up with the same idea.

If I had to guess, I'd say that this isn't due to any erroneous coding on the part of the teams involved. It's just smart to eliminate the possibility with certainty.

Let me know how it goes, and thanks for all the info here.

#31 I am not a coder, but I'll talk anyway in case my 'outside the square' viewpoint helps - if the vehicle image and the particles are not synced up then they'd both be using different methods of getting position. This sounds silly though, an emitter should be tied to the "image" of the vehicle even if the bounding box is some place else. You wouldn't want the server sending the vehicle position and emitter positions to the clients when the clients can just extrapolate the emitter position from the vehicle position to save on bandwidth and visual anomalies.

Or the particle blocks are using a 'velocity inheritance' which is applying the vehicle object velocity to the emitter position and getting out of sync because the 'image' is using a different method of predicition.

I do remember a $global setting which worked in T2 and torque which changes the netcode prediction for the clients so they see positions in a different way (i think it is different to the interpolate settings, but maybe not), so you see things more Tribes 1 style prediction than Tribes 2 - ie when a client lags in T2 they freeze and everyone sees them freeze solid in mid-air etc, but in Tribes 1 the server keeps moving them and everyone sees them continue to move. There is also the interpolation settings (used by chaingunners in T2 to get better aiming) so you exact positions with less interpolation. That is about all I know though, I can't seem to find my old notes.

#32 A possibility for the AI lag is that the AI isn't propagatings its velocity across the network properly, causing its model to continuously lag behind the real position. This would be consistent with the observed behavior (ie, speed-induced lag).

The issue with hopping might be caused by inaccurate client side predictions. Again, check that state is being replicated properly.

In fact, the root cause here might be that all the teams have slightly borked their vehicle code while tweaking, each in a different way, but enough to cause similar network accuracy errors.

One thing you might want to do is create some extra variables and render the last net update for each shape, to see what data is coming across the wire. It might be you need to up precision for the vehicles.

The packet size/ratea issues imply that the server is having a hard time fitting updates into the available bandwidth. How much data are you burning on each update of the vehicles? If you want to use 500 byte packets and you expect to have 10 vehicles on screen at a time, you're going to need to use less than 50 bytes/update on average.

You might also try adding some more prediction code to the vehicles, to catch "hopping" cases... or you could try making the shocks higher in general, or doing client side simulation with softer shock settings so that vehicles have a much harder time inadvertantly jumping.

Again, plotting the client side interpolation and net updates is going to be crucial to figuring this out.

#33 Couple things . . .

First, how do you figure out how many bytes you're sending per update?

Second, I don't think the shocks have much to do with the "hopping." Also, I kinda shy away from calling it "hopping" as it's a lot more like random translocation . . . that is if we're all talking about the same thing. I started the thread, but I have to admit I'm having a hard time following it and have had VERY little time to follow up on it. I hope to have time soon to preform some of the testing on HEAD that you guys were requesting. I'm fairly certain I could reproduce the actions.

[edit: no longer linking to our prototype]

#34 I usually figure a rule of thumb estimate by looking at my pack/unpackUpdate code and seeing what it's writing. Just add up the bits and divide by eight...

A random translocation... could be a warp, again. I'd enable as much debug state rendering code as possible here.

#35 Okay, pardon my ignorance . . . I'm pretty much clueless when it comes to debugging. Can someone tell me where I can look for information on how to enable that debug state rendering code as well as how I can enable the profiler and whatever other fun debug tools are available for use on the engine?

#36 I had another look at this over the weekend - It seems to be a problem with client side prediction.

If I disable client side prediction (see below), the 'hops' disappear completely. The ai movement is still not perfect, as it is always warping, so the movement is a bit jerky, but still a big improvement.

// in vehicle.cc, function processTick

if (isGhost()) {
   // If we haven't run out of prediction time,
   // predict using the last known move.
   if (mPredictionCount-- <= 0)
      return;
   move = &mDelta.move;

   ////////////////////////////////////////
   // Added this line to prevent prediction
   ////////////////////////////////////////
   return;
}

I have logged and plotted the server object's Z position versus the client Z position, and every so often, when doing client side prediction, there is a spike in the client's Z position. When the client is warping to match the server, the Z positions match fairly closely.

Any ideas?

#37 Nice work, Chris.

Have you tried plotting collision event against your Z plots? My theory is that the client prediction is inadvertantly colliding the ground and "bouncing" up, which is why I suggested earlier mucking with the client side modelling of the shocks.

Or it could be something weirder, but I think we're on our way to figuring it out.

#38 Hmm.

Interesting.

Just out of curisosity, is the tensor of inertia being updated after an unpackData? Does it need to be? (Looking in Vehicle)

#39 Dave, after looking at it, the instructions Mathieu sent just appear to reproduce the collision problems, not the warping and particle issues. We are most interested in examining the warping issues, since they are not expected at all.


If someone can reproduce the warping behavior from a clean HEAD get (or something with very minimal changes), we can certainly take a look. AFAIK, we've never observed that behavior with the base code. If anyone is able to reproduce the behavior from nearly clean code, Tim would like a journal (using the journaling functionality of the engine) of what happens. Please email Tim or I with all relevant instructions for code changes and the journal, if you can.

The particle stuff would be worth looking into as well, but after some initial investigations, this behavior appears to be produced by faulty collision boxes (see the comment by Robert Blanchet in Mathieu's .plan from a couple days ago). I don't think this conclusion is certain yet, though... Ben, Alex, and Robert will know more about the issue. If it turns out to be some kind of problem with the engine, we'll certainly try to look into it more.

The next step on the road to finding out what's going on with all this is to get a reproduction of the warping or particle issues from a (fairly) clean HEAD with a journal. We have not observed the behavior in our own tests yet, so we need someone to explain how to reproduce it.

If Mathieu's instructions did indicate how to reproduce the warping behavior, my apologies for missing it. I didn't observe the warping behavior from following the directions though.

Ok, keep us posted. :)

#40 We haven't had the issue pop up on us yet since up till this point we've only had hovers and flyers in game. Hopefully within 2 weeks we will have a functioning wheeled vehcile as well. Once we do i'm going to defintly keep an eye out for this and see if I can catch a journal of it.