Networked Terrain Deformation Problem
by Gary Preston · in Torque Game Engine · 09/12/2006 (5:57 am) · 26 replies
The following resource www.garagegames.com/index.php?sec=mg&mod=resource&page=view&qid=7343 covers terrain deformation using deformer objects to allow for networked play. I'm running a different version to this resource but experiencing the same problem as Nicholas Sandow describes in the resource comments.
When a player fires a projectile at the ground and deforms the terrain, the host client sees the correct deformation and will collide correctly with it (eg can sit on the bottom of the crater). Any other connected clients however will see a crater that is perhaps half (rough guess) as deep, yet will still collide the same as the server and thus end up underneath the terrain from their pov but resting on the bottom of the crater from the host clients pov.
Or as Nicholas puts it:
Having spent last week looking into the above bug, I'm still not really any closer to a solution. Does anyone with a working knowledge of the terrain code have any suggestions as to why connected clients are rendering the deformations differently to the server, yet appear to be using the same "correct" collision data as the host client?
So far I've checked the following:-
1) As the host client renders and collides with the deformations correctly, yet any connected clients do not, I thought this could be a problem with the networking side, with the host client seeing things correctly due to the networking "short circuits". However, I've checked the data transmit to each client by the deformation object's pack/unpack and it's the same as the data used/sent by the server.
2) The code looks like it should be deterministic and the log info on both client and server suggest this is the case. I've also implemented just the deformation code in a clean TGE1.4 to ensure the rest of the terrain modifcations made are not having a knock on effect.
Could it be that the terrain deformer object when processing on the server is updating some other structure used by the terrain code for rendering which isn't currently network enabled and thus only the short circuited host is seeing the correct deform. I'm a little stuck as to what this could be though, reading/tracing through the terrain source so far hasn't highlighted anything obvious.
The code makes a call to updateGrid which I read in another forum thread was created for working with the terrain editors and may not be suitable for a networked environment, but again reading through this doesn't suggest it's doing anything that would cause the above problem.
I've resolved the other problems with the resource such as forcing shapebase objects to update their working collision set to handle the case of terrain deforming directly underneath them but this one has got me stumped. My knowledge of the terrain code is very limited so if anyone has any insight/suggestions as to where to focus my attention I'd appreciate it.
If you need any more information, let me know.
When a player fires a projectile at the ground and deforms the terrain, the host client sees the correct deformation and will collide correctly with it (eg can sit on the bottom of the crater). Any other connected clients however will see a crater that is perhaps half (rough guess) as deep, yet will still collide the same as the server and thus end up underneath the terrain from their pov but resting on the bottom of the crater from the host clients pov.
Or as Nicholas puts it:
Quote:
The client does in fact see holes appear, but they are not drawn deep enough. In other words, when the client walks across a hole, the avatar will descend approximately the same amount as the server player, but will be drawn with half or all of his body in the ground.
There is no such problem for the server player. He sees the avatars positioned correctly on top of the terrain at the bottom of the hole.
Having spent last week looking into the above bug, I'm still not really any closer to a solution. Does anyone with a working knowledge of the terrain code have any suggestions as to why connected clients are rendering the deformations differently to the server, yet appear to be using the same "correct" collision data as the host client?
So far I've checked the following:-
1) As the host client renders and collides with the deformations correctly, yet any connected clients do not, I thought this could be a problem with the networking side, with the host client seeing things correctly due to the networking "short circuits". However, I've checked the data transmit to each client by the deformation object's pack/unpack and it's the same as the data used/sent by the server.
2) The code looks like it should be deterministic and the log info on both client and server suggest this is the case. I've also implemented just the deformation code in a clean TGE1.4 to ensure the rest of the terrain modifcations made are not having a knock on effect.
Could it be that the terrain deformer object when processing on the server is updating some other structure used by the terrain code for rendering which isn't currently network enabled and thus only the short circuited host is seeing the correct deform. I'm a little stuck as to what this could be though, reading/tracing through the terrain source so far hasn't highlighted anything obvious.
The code makes a call to updateGrid which I read in another forum thread was created for working with the terrain editors and may not be suitable for a networked environment, but again reading through this doesn't suggest it's doing anything that would cause the above problem.
I've resolved the other problems with the resource such as forcing shapebase objects to update their working collision set to handle the case of terrain deforming directly underneath them but this one has got me stumped. My knowledge of the terrain code is very limited so if anyone has any insight/suggestions as to where to focus my attention I'd appreciate it.
If you need any more information, let me know.
#2
A little theory behind the various terrain deformation resources out there:
--basically, the terrain itself isn't networked (although it is set to scopeAlways, this is simply to make sure the client never "throws it away").
--the operating principle behind the resource is that since the terrain itself isn't networked, we create a deformation object, and network that, and then apply the information within the deformation object to both the server's terrain (for authoritative collision information), and the client's terrain (for non-authoritative collision, as well as rendering).
Some troubleshooting questions:
You mention that the client seeing the bug sees himself basically underneath the terrain he sees, which implies one of two things:
--the server is constantly sending the update to the client saying "no, you are here", and the client is interpolating from his current expected position (due to client terrain out of snych with server), which should have a visual effect of "high speed jitter"--in other words, does your client see his avatar constantly jumping up, and then drifting down, over and over again?
-- you mention that you are updating the working collision set, but didn't say where--are you doing it on the server, the client, or both?
09/12/2006 (11:16 am)
It sounds as if your client's aren't fully applying the deformation to their terrain (which is critical).A little theory behind the various terrain deformation resources out there:
--basically, the terrain itself isn't networked (although it is set to scopeAlways, this is simply to make sure the client never "throws it away").
--the operating principle behind the resource is that since the terrain itself isn't networked, we create a deformation object, and network that, and then apply the information within the deformation object to both the server's terrain (for authoritative collision information), and the client's terrain (for non-authoritative collision, as well as rendering).
Some troubleshooting questions:
You mention that the client seeing the bug sees himself basically underneath the terrain he sees, which implies one of two things:
--the server is constantly sending the update to the client saying "no, you are here", and the client is interpolating from his current expected position (due to client terrain out of snych with server), which should have a visual effect of "high speed jitter"--in other words, does your client see his avatar constantly jumping up, and then drifting down, over and over again?
-- you mention that you are updating the working collision set, but didn't say where--are you doing it on the server, the client, or both?
#3
The terrain deformer after been created on the server, is ghosted across to the client and does its processing. I've checked this via a few console debug prints on the client.
The terrain on connected clients, does deform although only half (roughly) as deep as on the host client. Which suggests the terrain deformer object is been ghosted over. I've also checked the values ghosted over to make sure they're correct.
The working collision set update occurs both on client and server. The cache in question is the one held by shapebases. Without this, if the terrain deforms under the player, it does not recognise this, as it's still using the cached terrain data which causes all rayscasts/collision checks against the terrain to return that the player is already resting on the ground. It's only when the player moves a slight amount that the working cache is invalidated, so I added a bit of code to force this in the vacinity of the terrain deformer objects.
However just in case my "fix" was the cause of this problem, I applied the terrain resource to a clean tge1.4 and experienced the same problem.
Yes, there is a slight jitter on the connected client as though the servers idea of where the player should be differs from the clients. Which makes sense since the clients idea of the terrain is a crater say 1m deep where as the server is placing the player at the bottom of its crater which is 2m deep.
What I don't quite understand is how the terrain deformer applies on the server, yet once ghosted to the client fails to fully apply. Unless a value it requires is either not been tranmit to the client at all, or is losing precision in the transfer. I have checked for the latter, but not the former, I'll do so in the morning.
As an additional note, having done a bit more testing, a couple of times I've seen the terrain deform on the client in a slightly different position to that of the server, as a guess 30-40 units away.
I'm not sure if this related, but another bug I've been seeing is just after the game starts (prior to any terrain deforming) 3 pickups are spawned on the server. These are spawned via a ray cast at a random x/y from a height of 500 to determine the terrain height at that position and then placed at z+1. On the host client the three pickups appear just above the terrain as expected. On any other connected clients however, they appear underneath the terrain.
Spawning the objects from Z+40 however and allowing them time to fall to the ground will result in all 3 been correctly placed just above the terrain on both connected clients and the host client.
Anyhow, I'll have a more detailed look into the data been sent to the clients and that been received as well as checking whether any additional values are used by the terrain deformer class on the server that are not included in the pack updates.
I've not tried that, I'll give it a go tomorrow, although I'm going to hazard a guess that each client that connects will see half height craters and the server will believe the craters are much deeper.
09/12/2006 (12:04 pm)
Quote:
It sounds as if your client's aren't fully applying the deformation to their terrain (which is critical).
The terrain deformer after been created on the server, is ghosted across to the client and does its processing. I've checked this via a few console debug prints on the client.
The terrain on connected clients, does deform although only half (roughly) as deep as on the host client. Which suggests the terrain deformer object is been ghosted over. I've also checked the values ghosted over to make sure they're correct.
Quote:-- you mention that you are updating the working collision set, but didn't say where--are you doing it on the server, the client, or both?
The working collision set update occurs both on client and server. The cache in question is the one held by shapebases. Without this, if the terrain deforms under the player, it does not recognise this, as it's still using the cached terrain data which causes all rayscasts/collision checks against the terrain to return that the player is already resting on the ground. It's only when the player moves a slight amount that the working cache is invalidated, so I added a bit of code to force this in the vacinity of the terrain deformer objects.
However just in case my "fix" was the cause of this problem, I applied the terrain resource to a clean tge1.4 and experienced the same problem.
Quote:
--the server is constantly sending the update to the client saying "no, you are here", and the client is interpolating from his current expected position (due to client terrain out of snych with server), which should have a visual effect of "high speed jitter"--in other words, does your client see his avatar constantly jumping up, and then drifting down, over and over again?
Yes, there is a slight jitter on the connected client as though the servers idea of where the player should be differs from the clients. Which makes sense since the clients idea of the terrain is a crater say 1m deep where as the server is placing the player at the bottom of its crater which is 2m deep.
What I don't quite understand is how the terrain deformer applies on the server, yet once ghosted to the client fails to fully apply. Unless a value it requires is either not been tranmit to the client at all, or is losing precision in the transfer. I have checked for the latter, but not the former, I'll do so in the morning.
As an additional note, having done a bit more testing, a couple of times I've seen the terrain deform on the client in a slightly different position to that of the server, as a guess 30-40 units away.
I'm not sure if this related, but another bug I've been seeing is just after the game starts (prior to any terrain deforming) 3 pickups are spawned on the server. These are spawned via a ray cast at a random x/y from a height of 500 to determine the terrain height at that position and then placed at z+1. On the host client the three pickups appear just above the terrain as expected. On any other connected clients however, they appear underneath the terrain.
Spawning the objects from Z+40 however and allowing them time to fall to the ground will result in all 3 been correctly placed just above the terrain on both connected clients and the host client.
Anyhow, I'll have a more detailed look into the data been sent to the clients and that been received as well as checking whether any additional values are used by the terrain deformer class on the server that are not included in the pack updates.
Quote:What happens when you run a dedicated server, so only players in full client mode can connect?
I've not tried that, I'll give it a go tomorrow, although I'm going to hazard a guess that each client that connects will see half height craters and the server will believe the craters are much deeper.
#4
Honestly not sure, but it's definitely not applying the same deformation on server and client if you are seeing the "jump up, drift down" artifact happening--that's a sure symptom of client side positional de-synch.
09/12/2006 (12:12 pm)
It sounds as if (guessing at this point) that you are applying the deformation object before it's fully ghosted over, or somehow not getting the correct algorithm on the client to be used.Honestly not sure, but it's definitely not applying the same deformation on server and client if you are seeing the "jump up, drift down" artifact happening--that's a sure symptom of client side positional de-synch.
#5
Thanks for the pointers so far.
09/12/2006 (12:16 pm)
Yes, I think it's worth me spending a bit of time tommorrow checking exactly what is variables are been used by the deform code and whether these are all ghosted to the client correctly and available before the ghosted deform object starts to adjust the terrain.Thanks for the pointers so far.
#6
The results showed that the datablocks were fine, all data required was been sent and received correctly. However, the terrain deformer object was another matter. Although the data sent did match the data received, it became apparent the wrong data was been sent in some cases as the following snippet of the server-terrain.log shows
So it would appear that the reason the terrain deforms on the client in a different position to the server is that the initial update isn't sending a valid position. I'm not sure if this is perhaps a race condition? Since several times the ordering was different to the above. For example, in some cases processTick on the server occured, then the packUpdates for the networked clietn and locally connected clients occured, which resulted in a valid position.
I'll probably resolve this by sending much less info in the initialupdate and instead sending it via a separate updateMask which is only set once the server has completed the deformation in processTick (obviously this will only work for instant deforms, but that is all I need) that way position will definatly be valid.
However, this didn't fully explain the target height problem. But, pauls suggestion to try with a dedicated server and the above log files put me on the right track although I never got as far as testing with a dedicated server.
Looking at the log shows the server deforms the terrain during processTick, after which we have a packUpdate call for its locally connected client. The local client unpacks the data and then during processTick will deform the terrain. BUT as local clients are short circuited, this results in the servers terrain been deformed twice.
I've resolved this by checking in the activate deformation whether we have a client object AND if it's a local client connection, in which case I skip applying the deformation since the server has already processed it. So, the height bug is now fixed, I just need to rework the code to resolve the position bug :)
Big thanks to Stephen and Paul you've both helped a great deal, I think I had gotten to the point that I couldn't see the woods for the trees.
09/13/2006 (8:21 am)
Quick update, based on Stephen's comments about a networking sync problem I created a log just for terrain events and output all the variables values in the datablock packData/unpackData as well as the terrain deformers packUpdate and unpackUpdate.The results showed that the datablocks were fine, all data required was been sent and received correctly. However, the terrain deformer object was another matter. Although the data sent did match the data received, it became apparent the wrong data was been sent in some cases as the following snippet of the server-terrain.log shows
! Server is ghosting to the networked client
TerrainDeformer::packUpdate BEGIN(ID/GhostIndex 3471/-1)
InitialUpdateMask true
Pos X/Y/Z = 0.000000/0.000000/0.000000
mXSize = 0
mYSize = 0
mRadius = 42.000000
mDepth = 17.000000
mTargetHeight = 0.000000
mDeformShape = 2
mConformDistance = 21.000000
mInProgress = 1
mCurProgressX = 0.000000
mCurProgressY = 0.000000
ProgressMask true
mCurProgressX = 0.000000
mCurProgressY = 0.000000
mInProgress = 1
TerrainDeformer::packUpdate END
! Server is processing terrain deform, position is now valid, mTargetHeight is set in processTick!!!!
TerrainDeformer::processTick BEGIN
TerrainDeformer::activateDeformation BEGINDUMP
Pos X/Y/Z = -16.699100/-160.473999/90.014503
mXSize = 0
mYSize = 0
mRadius = 42.000000
mDepth = 17.000000
mTargetHeight = 0.000000
mDeformShape = 2
mConformDistance = 21.000000
mInProgress = 1
mCurProgressX = 0.000000
mCurProgressY = 0.000000
TerrainDeformer::activateDeformation ENDDUMP
TerrainDeformer::processTick END
! Server is ghosting deformation object to its locally connected client
TerrainDeformer::packUpdate BEGIN(ID/GhostIndex 3471/-1)
InitialUpdateMask true
Pos X/Y/Z = -16.699100/-160.473999/90.263885
mXSize = 0
mYSize = 0
mRadius = 42.000000
mDepth = 17.000000
mTargetHeight = 90.263885
mDeformShape = 2
mConformDistance = 21.000000
mInProgress = 1
mCurProgressX = 0.000000
mCurProgressY = 0.000000
ProgressMask true
mCurProgressX = 0.000000
mCurProgressY = 0.000000
mInProgress = 1
TerrainDeformer::packUpdate ENDSo it would appear that the reason the terrain deforms on the client in a different position to the server is that the initial update isn't sending a valid position. I'm not sure if this is perhaps a race condition? Since several times the ordering was different to the above. For example, in some cases processTick on the server occured, then the packUpdates for the networked clietn and locally connected clients occured, which resulted in a valid position.
I'll probably resolve this by sending much less info in the initialupdate and instead sending it via a separate updateMask which is only set once the server has completed the deformation in processTick (obviously this will only work for instant deforms, but that is all I need) that way position will definatly be valid.
However, this didn't fully explain the target height problem. But, pauls suggestion to try with a dedicated server and the above log files put me on the right track although I never got as far as testing with a dedicated server.
Looking at the log shows the server deforms the terrain during processTick, after which we have a packUpdate call for its locally connected client. The local client unpacks the data and then during processTick will deform the terrain. BUT as local clients are short circuited, this results in the servers terrain been deformed twice.
I've resolved this by checking in the activate deformation whether we have a client object AND if it's a local client connection, in which case I skip applying the deformation since the server has already processed it. So, the height bug is now fixed, I just need to rework the code to resolve the position bug :)
Big thanks to Stephen and Paul you've both helped a great deal, I think I had gotten to the point that I couldn't see the woods for the trees.
#7
09/13/2006 (11:51 am)
Wow Gary! I am impressed and can't wait to see what you come up with.
#8
I did this because I actually had deformations over time (peons levelling terrain so buildings could be placed), and other than a few minor issues that got resolved via good debugging output, it worked pretty well.
You may want to consider either destroying the ghost, or at least marking it as processTick(false) once the deformation is confirmed complete, otherwise you'll be stacking up unused derformation objects on the client (note that in most cases once the deformation is applied, you don't actually need it), but if you have a game where folks can re-join and/or join at a later time, you'll want the deformations to hang around so their terrain is in synch--simply a matter of design requirements.
09/13/2006 (12:34 pm)
@Gary: that sounds very similar to the direction I wound up going quite a while ago--my initial update mask was actually completely empty (just some info on the deformation size), and then further updates would be applied when received.I did this because I actually had deformations over time (peons levelling terrain so buildings could be placed), and other than a few minor issues that got resolved via good debugging output, it worked pretty well.
You may want to consider either destroying the ghost, or at least marking it as processTick(false) once the deformation is confirmed complete, otherwise you'll be stacking up unused derformation objects on the client (note that in most cases once the deformation is applied, you don't actually need it), but if you have a game where folks can re-join and/or join at a later time, you'll want the deformations to hang around so their terrain is in synch--simply a matter of design requirements.
#9
09/13/2006 (1:17 pm)
We need to keep the deformer objects hanging around on the server to handle the case of new players joining mid game, but I hadn't thought about destroying the ghosted deformer objects once finished with on the client. I might look into that.
#10
So, I take it the position problem applies to the stock terrain resource? I haven't observed that problem, so there is a chance (I guess) that this is limited to your code which I understand is somewhat different to the stock resource.
I will test your fix for the height problem. Thanks very much for resolving this.
09/17/2006 (10:34 pm)
Great stuff, Gary!So, I take it the position problem applies to the stock terrain resource? I haven't observed that problem, so there is a chance (I guess) that this is limited to your code which I understand is somewhat different to the stock resource.
I will test your fix for the height problem. Thanks very much for resolving this.
#11
09/18/2006 (3:58 am)
Nicholas: Yes, I think the position bug was related to our code. Hope the above fix helps you with the height problem :) 
#12
11/29/2006 (10:53 am)
If the MMO server has to be rebooted, do all the deformations get re-applied on reboot?
#13
11/29/2006 (11:02 am)
No. If you want it to do that you will have to modify the code to make it actually change the terrain file, save it as a new one(in case you ever want to reset everything) and have the server use the modified terrain file. Yoiu will also have to make sure that all clients get that terrain file.
#14
12/12/2006 (9:02 am)
Hey Gary had any luck so far?
#15
You can see the final version of all the various tweaks in Shelled! www.shelledgame.com :)
12/12/2006 (9:33 am)
Ron: If you read up a few posts, the one that starts "Quick update" has the root cause of the problem as well as the solution :) You can see the final version of all the various tweaks in Shelled! www.shelledgame.com :)
#16
12/12/2006 (9:39 am)
Cool Gary, would you mind sharing the updated code? I have read what you said to, but I have no idea where to start. BY the way I tried Shelled! and yup you definitely fixed and the game totally rocks man.
#17
I'll drop josh an email though and once I get a spare moment I'll look into knocking out a new resource. Can't make any promises on when that will be though.
12/12/2006 (12:20 pm)
I'd have to have a word with Josh before releasing any code. Although since we've made several changes across the engine to get everything working the way we wanted, it may take a while to isolate the changes. I'll drop josh an email though and once I get a spare moment I'll look into knocking out a new resource. Can't make any promises on when that will be though.
#18
12/12/2006 (12:27 pm)
Thank you Gary its really appreciated. 
#19
10/15/2007 (2:21 pm)
Any updates on this?
#20
Foot print and tire track decals remain floating.
It messes up player particle emmisions (dust, water, splash, ect.) for about a second.
Other than that, the network problem is fixed.
10/15/2007 (5:50 pm)
Well Adam Gary helped me out alot and I have a working network system with only a couple of bugs that I have yet to work out. Foot print and tire track decals remain floating.
It messes up player particle emmisions (dust, water, splash, ect.) for about a second.
Other than that, the network problem is fixed.
Torque Owner Paul /*Wedge*/ DElia