Ghost update packets: guaranteed delivery or not?
by Manoel Neto · in Torque 3D Professional · 11/03/2009 (2:06 pm) · 10 replies
The project I'm working requires a very different kind of player class. I reached the point where the current network update method, which sends absolute states like position, rotation, animation, etc isn't suitable, due to a bunch of animation features that are constantly upping the packet size and cannot be properly synchronized using the current method.
Since all my player's actions are triggered by sequences of inputs, I'm moving towards a system where the players send input sequence data to their ghosts and these ghosts, via C++ code and script callbacks, perform actions based on said inputs, hopefully reproducing what their server-counterparts did.
I'm aware of lag-related issues and I'm preparing for them, but this will fall apart pretty quickly if ghost updates aren't guaranteed delivery: a missing input packet will cause clients to lose sync with the server. I know event data (commandToServer and commandToClient) are guaranteed, and they are written in the same packet as ghost updates, but is the ghost data guaranteed as well, or do I need to setup an ack system for my input-based updates similar to the one used by the moves?
Since all my player's actions are triggered by sequences of inputs, I'm moving towards a system where the players send input sequence data to their ghosts and these ghosts, via C++ code and script callbacks, perform actions based on said inputs, hopefully reproducing what their server-counterparts did.
I'm aware of lag-related issues and I'm preparing for them, but this will fall apart pretty quickly if ghost updates aren't guaranteed delivery: a missing input packet will cause clients to lose sync with the server. I know event data (commandToServer and commandToClient) are guaranteed, and they are written in the same packet as ghost updates, but is the ghost data guaranteed as well, or do I need to setup an ack system for my input-based updates similar to the one used by the moves?
About the author
Recent Threads
#2
If a packet is dropped (on some reason),it will be resend again.
For what i know, readpacketdata/writepacketdata are very secure delivery mechanism,therefore they are used for the most important objects,like control objects.
11/03/2009 (2:39 pm)
Ghost updates should be guaranteed.If a packet is dropped (on some reason),it will be resend again.
For what i know, readpacketdata/writepacketdata are very secure delivery mechanism,therefore they are used for the most important objects,like control objects.
#3
i'm sure you know this, manoel, but the ghosting mechanism is really better-suited for sending state rather than events which imply state.
this is because you can get multiple events in the same channel for a given ghost for a given client. for example, this code (edited for clarity) from shapeBase has an obvious problem which has been discussed in the resources section for years now:
so if you do myPlayer.setSkinName("red.hat") followed immediately by myPlayer.setSkinName("red.boots") then the red.hat event is going to get lost.
the other advantage to sending the total state instead of the instructions for creating that state is that clients go out of scope or disconnect/reconnect, etc. i'm sure you're aware of this but thought i'd throw it out there.
is it possible to abstract or package your data differently so that it becomes feasible to send the entire state ?
for example, instead of sending commands to set the hat & boot colors to red, i defined a list of clothing items which the clients are aware of and introduced "outfitMask", which ghosts down a single smallish integer for each of the eight or ten clothing items, and the client looks up what the textures should be. more or less.
11/03/2009 (3:44 pm)
i've done lots of networking w/ ghosts and never seen a ghost update get lost, but i am also under the impression that they are not guaranteed. it might be possible to set the guaranteed-or-not-ness of a particular object.i'm sure you know this, manoel, but the ghosting mechanism is really better-suited for sending state rather than events which imply state.
this is because you can get multiple events in the same channel for a given ghost for a given client. for example, this code (edited for clarity) from shapeBase has an obvious problem which has been discussed in the resources section for years now:
ShapeBase::setSkinName(const char* name)
{
mSkinNameHandle = NetStringHandle(name);
setMaskBits(SkinMask);
}
.. and then in packupdate, we have:
if (stream->writeFlag(mask & SkinMask))
con->packNetStringHandleU(stream, mSkinNameHandle);so if you do myPlayer.setSkinName("red.hat") followed immediately by myPlayer.setSkinName("red.boots") then the red.hat event is going to get lost.
the other advantage to sending the total state instead of the instructions for creating that state is that clients go out of scope or disconnect/reconnect, etc. i'm sure you're aware of this but thought i'd throw it out there.
is it possible to abstract or package your data differently so that it becomes feasible to send the entire state ?
for example, instead of sending commands to set the hat & boot colors to red, i defined a list of clothing items which the clients are aware of and introduced "outfitMask", which ghosts down a single smallish integer for each of the eight or ten clothing items, and the client looks up what the textures should be. more or less.
#4
I can't go into much details, but the game I'm working on is way too dissimilar from a multiplayer shooter to the point where the complete state is becoming too big to synchronize properly. Specifically, I have complex interactions between actors that involve animations and movements performed in sequence which must be accurate in a way that is a bit too much for the current network model to handle. I also have a quite complex control layer, with several states and possible context-sensitive movements, which is currently driven by script on the server.
My plan is to have the control layer send inputs from the server to the client (either inputs sent from clients, or inputs generated by AI), and have these inputs fire the control layer state-based callbacks on both sides. There are other measures I'm taking to make everything run in sync, which is another problem with using the current model on my project - I cannot have the server advance the simulation without having the necessary moves from all clients for that simulation step, since it would cause extreme unfairness.
This way I will "lock" the server-to-client packet into a fixed size, and can go as overboard as possible with my animation system.
This is more or less the way many RTS games perform their networking, and is also how fighting games do it.
(I still plan to keep state updates around for emergency synchronization situations).
11/03/2009 (4:15 pm)
Thanks for the info, it helps a lot :)I can't go into much details, but the game I'm working on is way too dissimilar from a multiplayer shooter to the point where the complete state is becoming too big to synchronize properly. Specifically, I have complex interactions between actors that involve animations and movements performed in sequence which must be accurate in a way that is a bit too much for the current network model to handle. I also have a quite complex control layer, with several states and possible context-sensitive movements, which is currently driven by script on the server.
My plan is to have the control layer send inputs from the server to the client (either inputs sent from clients, or inputs generated by AI), and have these inputs fire the control layer state-based callbacks on both sides. There are other measures I'm taking to make everything run in sync, which is another problem with using the current model on my project - I cannot have the server advance the simulation without having the necessary moves from all clients for that simulation step, since it would cause extreme unfairness.
This way I will "lock" the server-to-client packet into a fixed size, and can go as overboard as possible with my animation system.
This is more or less the way many RTS games perform their networking, and is also how fighting games do it.
(I still plan to keep state updates around for emergency synchronization situations).
#5
11/03/2009 (4:23 pm)
sounds like you're on it!
#6
There's also GGPO, which is a modified version of an arcade emulator with awesome netcode added to it. It sends and receives only inputs, and manages to offer excellent network play experience that is very latency-tolerant for games that were never designed with networking in mid.
The game I'm working on is played by few players (2-4) and is meant to be competitive. It's borderline P2P, but we'll use dedicated servers to prevent cheating (AI is run on the server) among other things. So the server must adjust the simulation so it is fair regardless of each individual player's latency. Right now if a player freezes, the server simply continues simulating giving huge advantage to the unlagged player.
11/03/2009 (4:33 pm)
There's a nice article on GamaSutra about how they did the networking in Age of Empires. Back then they had 8 players with 250 units each, playing on 33.6k modems. Sending full state updates was pretty much unfeasible back then, so they worked into sending orders and making sure they were properly ordered and executed synchronously.There's also GGPO, which is a modified version of an arcade emulator with awesome netcode added to it. It sends and receives only inputs, and manages to offer excellent network play experience that is very latency-tolerant for games that were never designed with networking in mid.
The game I'm working on is played by few players (2-4) and is meant to be competitive. It's borderline P2P, but we'll use dedicated servers to prevent cheating (AI is run on the server) among other things. So the server must adjust the simulation so it is fair regardless of each individual player's latency. Right now if a player freezes, the server simply continues simulating giving huge advantage to the unlagged player.
#7
But writePacketData/readPacketData (the networking used for the control object and the camera) that collects and sends the player command, is guaranteed to be delivered in the shortest possible time, include an ACK mechanism, and command are sends with highest priority in each packet sent, as long as the sender doesn't receive the ACK (so a command can be send several times if the ACK is a bit long to come, but that situation is handled correctly).
Nicolas Buquet
www.buquet-net.com/cv/
11/03/2009 (4:59 pm)
I don't think that packUpdate/unpackUpdate are guaranteed, because, I Orion said, a newer state can cancel an older state that is not send yet.But writePacketData/readPacketData (the networking used for the control object and the camera) that collects and sends the player command, is guaranteed to be delivered in the shortest possible time, include an ACK mechanism, and command are sends with highest priority in each packet sent, as long as the sender doesn't receive the ACK (so a command can be send several times if the ACK is a bit long to come, but that situation is handled correctly).
Nicolas Buquet
www.buquet-net.com/cv/
#8
In the NetEvent class, there are is a variable controlling the guarantee type of the command. The default is ordered and guaranteed but there are others you can choose from. The system definitely does its best to ensure that all clients recieve all of the updates.
11/03/2009 (5:10 pm)
Quote:I don't think that packUpdate/unpackUpdate are guaranteed, because, I Orion said, a newer state can cancel an older state that is not send yet.That doesn't imply that the system doesn't guarantee updates are received at all.
In the NetEvent class, there are is a variable controlling the guarantee type of the command. The default is ordered and guaranteed but there are others you can choose from. The system definitely does its best to ensure that all clients recieve all of the updates.
#9
Since the moveList implements an ACK system, to make sure all moves are sent in their proper sequence, I assume it is possible for a dropped packet to cause even a writePacketData() to be lost, and the moveList uses its ACK system to re-send moves until they are confirmed by the server.
Maybe instead of modifying pack/unpackUpdate(), it would be better to create a new write/read method to send input events from the server to the client with its own ACK system, like the moves do, and place it in a prioritized place in the packet (and make sure everything fits). I'd then keep pack/unpackUpdate() update around, but use it only for synchronization validation (since only a couple ghosts would be able to update each tick since they'd have less room in the package).
@Phillip: would you suggest using an NetEvent-derived class to send the input events? Sounds much more logical now.
11/03/2009 (5:15 pm)
Looking at GameConnection::writePacket(), I don't see any major distinction between write/readPacketData() and pack/unpackUpdate(), other than some ghosted objects not been given the chance to send their data if the current packet is full (they're the last one to get on the train). Since writePacketData is among the first things to go in the packet, it's never intentionally skipped.Since the moveList implements an ACK system, to make sure all moves are sent in their proper sequence, I assume it is possible for a dropped packet to cause even a writePacketData() to be lost, and the moveList uses its ACK system to re-send moves until they are confirmed by the server.
Maybe instead of modifying pack/unpackUpdate(), it would be better to create a new write/read method to send input events from the server to the client with its own ACK system, like the moves do, and place it in a prioritized place in the packet (and make sure everything fits). I'd then keep pack/unpackUpdate() update around, but use it only for synchronization validation (since only a couple ghosts would be able to update each tick since they'd have less room in the package).
@Phillip: would you suggest using an NetEvent-derived class to send the input events? Sounds much more logical now.
#10
11/03/2009 (5:39 pm)
I can't believe I forgot about NetEvent! Seems to be the ideal solution, and it even can be used to solve the problem of keeping the rest of the simulation in sync with the input system. We can wrap all our non-input state changes using NetEvents and make sure everything happens in the correct order in all clients as well as tying inputs to simulation ticks, and will make it very straightforward to maintain a buffer of updates ahead to keep the game going smoothly. 
Employee Scott Burns
GarageGames