Of Looting Corpses
by Dusty Monk · 10/14/2007 (9:24 pm) · 6 comments
Just going to drop a quick line of introduction here -- I'm a game developer that works for a major developer, and in my *off* time I love working with the Torque Engine just to see how much one person, working by themselves with Torque, is capable of developing. Like everyone else in the known world, I'm building an MMO. Okay, I'm not so naive as to think I can build an entire MMO. What I can do though is build a prototype. So I'm building an MMO prototype.
Now, this week I was working on looting corpses. I have loot being generated on the AI's when they die, and being replicated down to the client. And on the client side, I've added "CorpseObjectType" to my "work on object" filter, so the cursor turns into a little hand when you mouse over the corpse, and you can r-click on it. I even have a loot particle emitter added to the player class. When a player dies, loot is generated for him, and a flag on him is set to "isLootable". In the player "advanceTime" routine on the client, where we update all the other particle effects, we now call "upsateLootParticle" too, which checks the isLootable flag, turns on the particle system if it's not on, advances the particle system if it's already on, and if the isLootable flag is set to false, checks to see if the particle is running, and if so kills is.
So I got corpses dying, and cool loot sparkly's sparkling over them. My avatar is so thrilled she might be able to loot something soon. All is good.
Then.. this weekend, I sat down to get the server to actually *do* the work, and ran into a snag. See, when you r-click on the object on the client, I don't do the raycast on the client and send a request to loot to the server for the object, as you might expect. Instead, I send a generic "WorkOnObject" request to the server, and I send the [i]mouse coords[i] the player used to the server. Then I do the actual raycast on the server, and detect what type of object is under the mouse *server* side, and generate the appropriate command there. This is in general safer, and should help to keep you from generating false checks. In general, trust the client as little as possible.
So, the thing was, when I did the raycast against the corpse of the ai, the server was coming back empty! Nothing there! WTF! I have vehicles in the level. I use the same command to mount vehicles.. and the server check works for those, so I'm pretty sure it's not a problem in the casting of the ray line. Maybe the object type is not getting set correctly, or maybe CorpseObjectType isn't set correctly for AI players?
Well one thing I thought I would try is just seeing if I could get a positive raycast against an AIPlayer -- either alive *or* dead. So I went to add $TypeMasks::AIObjectTypeMask to my search mask. And.. DOH.. undefined! Apparently the guys down at GG didn't think you'd ever want to actually do a raycast for AI players. Actually.. this doesn't surprise me. If you look at AIPlayer, it's pretty clearly a very slim, one-off thing that I suspect was added late in the process. Definitly a second class citizen.
So the first thing I did was expose AIObjectType to the scripts. In main.cc, under the long list of setIntVariables, I added AIObjectType:
So I did the check, and sure enough, the raycast returned positive for the AIPlayer. Right up until he died. Then the player check failed for him too. That kind of surprised me.. because if you look in player::updateDamageStates, where we 'or-in' the corpse mask, and negate the player mask, we don't change anything with the aiplayer mask. So even after the player was dead.. I'd expect the raycast to work.
Finally, there was nothing left to do but roll up the sleeves and step deep into castRay, figure out what was going on. The console function didn't scare me.. after all it's just a thin wrapper for the fella that does the real work -- Container::castRay. This is the thing that draws a line across the grid system Torque uses for collision checks, and it looks inside the bin of objects it keeps in each bin, and then does a ray cast against them to see if you've hit them. Least, I'm pretty sure that' what it's doing. I haven't seen any resources or articles on the inner workings of Torque's collision checks, though I'm sure they're out there.
Fortunately, it wasn't necessary to fully understand the routine to find the bug. Because as we iterate over each object within the bin, once we determine that the line crosses the bounding box of the object, then we transform the end points of the line, and we call [bold]the object's raycast routine[/bold] to determine if our line actually hits. Which means, for players, and their children, AIPlayers, we're calling player::castRay. Now.. if you go and look at player::castRay.. guess what? The first thing it does is early out if you're not alive. DOH.
Just to make sure this wasn't something *I* added to player, I loaded up the default tge152 version of this module. Yap. It early outs if you're alive. So I removed that check.
I'm not too worried this is going to break things. It's obvious the guys at GG thought corpses want to be something you never collide with. I get that. But if you look throughout the code, there are lots of *other* things you don't ever want to collide with either. In fact they've made several copies of the special set of flags to mark these things. And that's how you *should* do it. Use the type system that you've built to exclude that kind of stuff -- not hard coded hacks of "oh if you're a corpse just return false from player::castRay."
Mystery solved. I now cast rays against corpses, get positives back, and can generate the appropriate routine. Sadly, my time's all up, so actually writing the code that loots the corps will hav to wait until next week. Besides, I gotta go build that stupid loot window UI before I can do anything.
Word.
Now, this week I was working on looting corpses. I have loot being generated on the AI's when they die, and being replicated down to the client. And on the client side, I've added "CorpseObjectType" to my "work on object" filter, so the cursor turns into a little hand when you mouse over the corpse, and you can r-click on it. I even have a loot particle emitter added to the player class. When a player dies, loot is generated for him, and a flag on him is set to "isLootable". In the player "advanceTime" routine on the client, where we update all the other particle effects, we now call "upsateLootParticle" too, which checks the isLootable flag, turns on the particle system if it's not on, advances the particle system if it's already on, and if the isLootable flag is set to false, checks to see if the particle is running, and if so kills is.
So I got corpses dying, and cool loot sparkly's sparkling over them. My avatar is so thrilled she might be able to loot something soon. All is good.
Then.. this weekend, I sat down to get the server to actually *do* the work, and ran into a snag. See, when you r-click on the object on the client, I don't do the raycast on the client and send a request to loot to the server for the object, as you might expect. Instead, I send a generic "WorkOnObject" request to the server, and I send the [i]mouse coords[i] the player used to the server. Then I do the actual raycast on the server, and detect what type of object is under the mouse *server* side, and generate the appropriate command there. This is in general safer, and should help to keep you from generating false checks. In general, trust the client as little as possible.
So, the thing was, when I did the raycast against the corpse of the ai, the server was coming back empty! Nothing there! WTF! I have vehicles in the level. I use the same command to mount vehicles.. and the server check works for those, so I'm pretty sure it's not a problem in the casting of the ray line. Maybe the object type is not getting set correctly, or maybe CorpseObjectType isn't set correctly for AI players?
Well one thing I thought I would try is just seeing if I could get a positive raycast against an AIPlayer -- either alive *or* dead. So I went to add $TypeMasks::AIObjectTypeMask to my search mask. And.. DOH.. undefined! Apparently the guys down at GG didn't think you'd ever want to actually do a raycast for AI players. Actually.. this doesn't surprise me. If you look at AIPlayer, it's pretty clearly a very slim, one-off thing that I suspect was added late in the process. Definitly a second class citizen.
So the first thing I did was expose AIObjectType to the scripts. In main.cc, under the long list of setIntVariables, I added AIObjectType:
Con::setIntVariable("$TypeMasks::DamagableItemObjectType", DamagableItemObjectType);
Con::setIntVariable("$TypeMasks::InteriorMapObjectType", InteriorMapObjectType);
// what do you have against AI's?
Con::setIntVariable("$TypeMasks::AIObjectType", AIObjectType);So I did the check, and sure enough, the raycast returned positive for the AIPlayer. Right up until he died. Then the player check failed for him too. That kind of surprised me.. because if you look in player::updateDamageStates, where we 'or-in' the corpse mask, and negate the player mask, we don't change anything with the aiplayer mask. So even after the player was dead.. I'd expect the raycast to work.
Finally, there was nothing left to do but roll up the sleeves and step deep into castRay, figure out what was going on. The console function didn't scare me.. after all it's just a thin wrapper for the fella that does the real work -- Container::castRay. This is the thing that draws a line across the grid system Torque uses for collision checks, and it looks inside the bin of objects it keeps in each bin, and then does a ray cast against them to see if you've hit them. Least, I'm pretty sure that' what it's doing. I haven't seen any resources or articles on the inner workings of Torque's collision checks, though I'm sure they're out there.
Fortunately, it wasn't necessary to fully understand the routine to find the bug. Because as we iterate over each object within the bin, once we determine that the line crosses the bounding box of the object, then we transform the end points of the line, and we call [bold]the object's raycast routine[/bold] to determine if our line actually hits. Which means, for players, and their children, AIPlayers, we're calling player::castRay. Now.. if you go and look at player::castRay.. guess what? The first thing it does is early out if you're not alive. DOH.
Just to make sure this wasn't something *I* added to player, I loaded up the default tge152 version of this module. Yap. It early outs if you're alive. So I removed that check.
bool Player::castRay(const Point3F &start, const Point3F &end, RayInfo* info)
{
// DW - 9/14/07 - Removing this. This effectively prevents you from
// ever colliding with a corpse, which you need to do if you want to,
// oh, say, loot him.
/*
if (getDamageState() != Enabled)
return false;
*/I'm not too worried this is going to break things. It's obvious the guys at GG thought corpses want to be something you never collide with. I get that. But if you look throughout the code, there are lots of *other* things you don't ever want to collide with either. In fact they've made several copies of the special set of flags to mark these things. And that's how you *should* do it. Use the type system that you've built to exclude that kind of stuff -- not hard coded hacks of "oh if you're a corpse just return false from player::castRay."
Mystery solved. I now cast rays against corpses, get positives back, and can generate the appropriate routine. Sadly, my time's all up, so actually writing the code that loots the corps will hav to wait until next week. Besides, I gotta go build that stupid loot window UI before I can do anything.
Word.
About the author
Dusty Monk is founder and president of Windstorm Studios, an independant game studio. Formerly a sr. programmer at Ensemble Studios, Dusty has worked on AAA titles such as Age of Empires II & III, and Halo Wars.
#2
10/15/2007 (1:37 am)
This was a very nice read, good job there Devon! 
#3
Couple of comments:
This happens often, and much of it is simply a factor of taking a commercial game's source code (Tribes) and turning it into an agnostic game engine (Torque). I can't argue your point at all--because you are correct! I can however comment on that specific issue:
The Player class is actually a reference implementation when it gets down to it--specifically, it's a trimmed down example of actual game code from Tribes. As a reference implementation, it contains quite a few optimizations, and in some cases flat out "hacks" to "get 'r done" so to speak. All I can guess is that one of the Tribes developers had a deadline, and/or wanted to optimize as much as humanly possible, and threw that early out in for one of those two reasons. It's certainly not best practice, but as you well know, sometimes it's just the way things happen!
I 1000% agree with your strategy of "trust(ing) the client as little as possible", and I applaud you for thinking this through completely--but I did want to point out a possible flaw in your implementation. By definition, a client's simulation and the server's authoritative simulation are never 100% in synchronization--the fact that you can never have a zero network delay, combined with the possibility that a particular client may be even further behind in synchronization/ghosting can lead to unusual cases where the client's simulation would indicate visually that a selection is valid, but the server would not match the same object. This is a both a short term (couple of network cycles most likely) and rare (only in high latency/packet loss scenarios) situation, but it could be a frustrating one for your end user if the server cast ray and the client's "player perception of a cast ray" differ.
A more involved strategy for this to both avoid the issue, as well as continue to not trust the client, might be to do your casts on the client, send up a "request for selection" to the server, and have the server simply validate the selection request within a threshold instead of actively performing the cast ray. Basically, a "client says he wants to select object XX, is that feasible?", and the server would authoritatively say ok/no.
This would also allow your client to at least simulate reacting to object selection rapidly in a high latency environment--if, for example, your particular client is getting 1000 ms latencies, it would take a round trip of 2+ seconds before the client is told that a selection occurred at all in your setup--and in very high latency situations, it gets worse. I know from personal experience playing many MMO's that this was incredibly frustrating--click on your corpse, but can't loot it (or at least appear to be looting it) for many seconds.
It's a subtle difference, but I think a good once to consider, at least from the planning perspective :)
10/15/2007 (8:18 am)
Good write up of your research and discovery, and thanks for sharing!Couple of comments:
Quote:
And that's how you *should* do it. Use the type system that you've built to exclude that kind of stuff -- not hard coded hacks of "oh if you're a corpse just return false from player::castRay."
This happens often, and much of it is simply a factor of taking a commercial game's source code (Tribes) and turning it into an agnostic game engine (Torque). I can't argue your point at all--because you are correct! I can however comment on that specific issue:
The Player class is actually a reference implementation when it gets down to it--specifically, it's a trimmed down example of actual game code from Tribes. As a reference implementation, it contains quite a few optimizations, and in some cases flat out "hacks" to "get 'r done" so to speak. All I can guess is that one of the Tribes developers had a deadline, and/or wanted to optimize as much as humanly possible, and threw that early out in for one of those two reasons. It's certainly not best practice, but as you well know, sometimes it's just the way things happen!
Quote:
See, when you r-click on the object on the client, I don't do the raycast on the client and send a request to loot to the server for the object, as you might expect. Instead, I send a generic "WorkOnObject" request to the server, and I send the mouse coords the player used to the server. Then I do the actual raycast on the server, and detect what type of object is under the mouse *server* side, and generate the appropriate command there. This is in general safer, and should help to keep you from generating false checks. In general, trust the client as little as possible.
I 1000% agree with your strategy of "trust(ing) the client as little as possible", and I applaud you for thinking this through completely--but I did want to point out a possible flaw in your implementation. By definition, a client's simulation and the server's authoritative simulation are never 100% in synchronization--the fact that you can never have a zero network delay, combined with the possibility that a particular client may be even further behind in synchronization/ghosting can lead to unusual cases where the client's simulation would indicate visually that a selection is valid, but the server would not match the same object. This is a both a short term (couple of network cycles most likely) and rare (only in high latency/packet loss scenarios) situation, but it could be a frustrating one for your end user if the server cast ray and the client's "player perception of a cast ray" differ.
A more involved strategy for this to both avoid the issue, as well as continue to not trust the client, might be to do your casts on the client, send up a "request for selection" to the server, and have the server simply validate the selection request within a threshold instead of actively performing the cast ray. Basically, a "client says he wants to select object XX, is that feasible?", and the server would authoritatively say ok/no.
This would also allow your client to at least simulate reacting to object selection rapidly in a high latency environment--if, for example, your particular client is getting 1000 ms latencies, it would take a round trip of 2+ seconds before the client is told that a selection occurred at all in your setup--and in very high latency situations, it gets worse. I know from personal experience playing many MMO's that this was incredibly frustrating--click on your corpse, but can't loot it (or at least appear to be looting it) for many seconds.
It's a subtle difference, but I think a good once to consider, at least from the planning perspective :)
#4
In our project our servers "never" trust the clients, but it's allowed to do some "hard work" on client-side.
Making raycast on client is good, and our server only say yes/no (like Stephen mentioned), but I also added additional checks to prevent "spamming" servers, before sending every command I check when the last command was sent.
Imagine someone constantly clicking all around - it could make a heavy load on network if you are sending "mouse coords" on every click.
I do similar to that:
You can set it to 500 ms, so only two commands in a second will be sent to server. But you need to think hard on using it, be sure the "required" or "important" commands are sent always.
Edit: oops, too early pressed the submit:)
10/15/2007 (8:52 am)
This is interesting.In our project our servers "never" trust the clients, but it's allowed to do some "hard work" on client-side.
Making raycast on client is good, and our server only say yes/no (like Stephen mentioned), but I also added additional checks to prevent "spamming" servers, before sending every command I check when the last command was sent.
Imagine someone constantly clicking all around - it could make a heavy load on network if you are sending "mouse coords" on every click.
I do similar to that:
if ( getSimTime() - $tmp::prevSimTime > $minDelayTime )
{
$tmp::prevSimTime = getSimTime();
commandToServer('targetAction', %param1, %param2);
}Where $minDelayTime = the delay in ms between commands.You can set it to 500 ms, so only two commands in a second will be sent to server. But you need to think hard on using it, be sure the "required" or "important" commands are sent always.
Edit: oops, too early pressed the submit:)
#5
We've all tried (okay I have, I'm guessing you have too) tried to click on corpses in WoW that our client said were clearly right in front of us, but the server didn't seem inclined to agree. I can also totally see including the anti-spam code that bank added as well. I think I might optimize this to disallow to checks within the time frame that were the same command, but to allow two checks of different commands, or different targets. This would require you caching on the client some notion of "lastCommandToServer", along with "lastObjectWorkedOn".
This would prevent the guy from spamming r-click on a particular target, but still allow you what are more likely to be legimate cases of switching targets quickly. Not sure, but as bank suggested, I'll definitely have to think about it some more.
Both excellent suggestions - thanks for taking the time to read and comment!
10/15/2007 (9:18 am)
I like that approach. The "select object" with a range specification. You'll still have to do the raycast on the server, or at least perform a proximity check between the client and the object you want to work on, but it does allow you to work around the high latency situations. I could even see a more evolved solution where you increase your range based on the last detected latency with the client, so perhaps in high latency situations you allow them more leeway on what's a valid check. Maybe. We've all tried (okay I have, I'm guessing you have too) tried to click on corpses in WoW that our client said were clearly right in front of us, but the server didn't seem inclined to agree. I can also totally see including the anti-spam code that bank added as well. I think I might optimize this to disallow to checks within the time frame that were the same command, but to allow two checks of different commands, or different targets. This would require you caching on the client some notion of "lastCommandToServer", along with "lastObjectWorkedOn".
This would prevent the guy from spamming r-click on a particular target, but still allow you what are more likely to be legimate cases of switching targets quickly. Not sure, but as bank suggested, I'll definitely have to think about it some more.
Both excellent suggestions - thanks for taking the time to read and comment!
#6
10/23/2007 (5:07 am)
Wicked, thank you! 
Torque 3D Owner Max