Torque Networking Speed
by Guimo · in Torque Game Engine · 12/12/2006 (2:59 pm) · 28 replies
Greetings,
Im programming my game but found some performace problems. Maybe you can give me some hints about how to solve them but first I would like to understand whats happening. I think its a problem with the networking model. Please help me understand.
Before anything, Im using TGE 1.4+TLK and compiling in DEBUG mode. I know this may hurt the performance but Im really not sure. I also run the game with the 'Host Multiplayer' starting option. Also I use a wireless network card (54g) that I suspect may be losing some packets du to the PC location forcing message repetition and losing a lot of speed in the process.
My game is a space fighting game like StarControl. The game allows you take some powerups like weapons and multishots (one weapon firing many rounds at once). Some weapons require fast firing and create many rounds each seconds. I.E. The machinegun fires 10 rounds each second. If you get the multiweapon you may fire 2, 3, 4 or even 5 shots each round forming an arch. Thats 50 projectiles each second!!!
So when the weapon is fully loaded I press the fire button. During the First 1 or 2 seconds I can see all the rounds being fired from the weapon. But after that the rounds take longer to 'appear' until they no longer appear at all.
I know Torque uses a strong client/server model, so when the ship fires, the server creates the projectile and that new object is propagated to all the clients. So I guess that as Im creating too many objects and the network becomes saturated. So messages take longer and longer to arrive and finally they take so long that when the client receives them the bullet is no longer alive so its not rendered.
Now, this answer would be ok for me except that Im running the client and server in the same PC, and its not a dedicated server, its the client running as server. I thougt that in this case Torque would use a 'direct' message transmission instead of using the network. If thats the case, then why Im losing those packets... maybe the Host Multiplayer forces the client to connect to the server in a networked solution?
I have other weapons like the spreadgun. When completely upgraded it fires a projectile for each 5 degrees in a 180 degrees arc each second (destruction yeah!!!). In this case I can even SEE the arc forming when each bullet appears.
I noticed the problem just starting with my game. I wanted to create a rotating minefield so I decided to create 2000 asteroids dynamically at startup. Unfortunately it just took TOO long (and the rendering was still worse) so I decided to use a Replicator even when the asteroid field is now just dynamic.
So I would like to know if you have found similar problems. I dont know if just connecting my PC with a LAN cable would solve anything... or compiling in release may help. I must host as Multiplayer because is a Lan game so thats out of question. If anybody can give me any ideas with this subject they are welcome.
Luck!
Guimo
Im programming my game but found some performace problems. Maybe you can give me some hints about how to solve them but first I would like to understand whats happening. I think its a problem with the networking model. Please help me understand.
Before anything, Im using TGE 1.4+TLK and compiling in DEBUG mode. I know this may hurt the performance but Im really not sure. I also run the game with the 'Host Multiplayer' starting option. Also I use a wireless network card (54g) that I suspect may be losing some packets du to the PC location forcing message repetition and losing a lot of speed in the process.
My game is a space fighting game like StarControl. The game allows you take some powerups like weapons and multishots (one weapon firing many rounds at once). Some weapons require fast firing and create many rounds each seconds. I.E. The machinegun fires 10 rounds each second. If you get the multiweapon you may fire 2, 3, 4 or even 5 shots each round forming an arch. Thats 50 projectiles each second!!!
So when the weapon is fully loaded I press the fire button. During the First 1 or 2 seconds I can see all the rounds being fired from the weapon. But after that the rounds take longer to 'appear' until they no longer appear at all.
I know Torque uses a strong client/server model, so when the ship fires, the server creates the projectile and that new object is propagated to all the clients. So I guess that as Im creating too many objects and the network becomes saturated. So messages take longer and longer to arrive and finally they take so long that when the client receives them the bullet is no longer alive so its not rendered.
Now, this answer would be ok for me except that Im running the client and server in the same PC, and its not a dedicated server, its the client running as server. I thougt that in this case Torque would use a 'direct' message transmission instead of using the network. If thats the case, then why Im losing those packets... maybe the Host Multiplayer forces the client to connect to the server in a networked solution?
I have other weapons like the spreadgun. When completely upgraded it fires a projectile for each 5 degrees in a 180 degrees arc each second (destruction yeah!!!). In this case I can even SEE the arc forming when each bullet appears.
I noticed the problem just starting with my game. I wanted to create a rotating minefield so I decided to create 2000 asteroids dynamically at startup. Unfortunately it just took TOO long (and the rendering was still worse) so I decided to use a Replicator even when the asteroid field is now just dynamic.
So I would like to know if you have found similar problems. I dont know if just connecting my PC with a LAN cable would solve anything... or compiling in release may help. I must host as Multiplayer because is a Lan game so thats out of question. If anybody can give me any ideas with this subject they are welcome.
Luck!
Guimo
About the author
#22
Thanks, I should have realized that the netgraph would give us this information. And, now that I understand that Torque always sends 200 bytes of information, this data just started making alot more sense to me.
One followup question, can someone explain how there can be 200 bytes sent, and 215 bytes recieved, as in your first example?
Happy Holidays!
Todd
12/24/2006 (11:18 am)
Guimo,Thanks, I should have realized that the netgraph would give us this information. And, now that I understand that Torque always sends 200 bytes of information, this data just started making alot more sense to me.
One followup question, can someone explain how there can be 200 bytes sent, and 215 bytes recieved, as in your first example?
Happy Holidays!
Todd
#23
a) Up to 200, per client. Each client may be getting different information based on scoping rules.
b) only if the client has the newly created object in scope. Additionally, the very first update will be larger per object (InitialUpdate) than follow on updates.
c) Correct, and will have higher priority in the packet than ghost updates (network events are written before ghost updates are run)
d) Not completely true. In fact, true projectiles (not flying vehicles like you for some reason made yours) actually have a quite high priority over other more "normal" ghosted objects. Object network priority is defined within the method ::getUpdatePriority(), implemented per object class. Of course, NetEvents get packed first, so they are "higher priority" than any ghosted object other than the control object.
e) Any updates not sent will be sent the next network tick, which is defined by your PacketRate. Stock is 200 ms, not 32 (32 is the physics processing frequency). Your derived assumption is however true, and in fact happens more frequently since the period is longer for network updates.
f) You should most probably be using the Projectile class, not FlyingVehicle class!
g) Possibly, but you are correct, shouldn't be affecting you.
h) Aye, very true if you want hundreds of projectiles that are all related/grouped within a cluster.
I'm not sure what you are defining latency as--but it still takes time to pack up the updates, and unpack them on the other side. I'm guessing that the 45-70 is part of that processing time, and not "network ping time" as you are probably defining latency.
I'm reasonably sure that Torque may over-pack, using whatever bytes are necessary to finish a particular packUpdate(), but then stop packing new objects into that particular packet.
12/24/2006 (3:52 pm)
Some minor corrections to your "what I learned" list:a) Up to 200, per client. Each client may be getting different information based on scoping rules.
b) only if the client has the newly created object in scope. Additionally, the very first update will be larger per object (InitialUpdate) than follow on updates.
c) Correct, and will have higher priority in the packet than ghost updates (network events are written before ghost updates are run)
d) Not completely true. In fact, true projectiles (not flying vehicles like you for some reason made yours) actually have a quite high priority over other more "normal" ghosted objects. Object network priority is defined within the method ::getUpdatePriority(), implemented per object class. Of course, NetEvents get packed first, so they are "higher priority" than any ghosted object other than the control object.
e) Any updates not sent will be sent the next network tick, which is defined by your PacketRate. Stock is 200 ms, not 32 (32 is the physics processing frequency). Your derived assumption is however true, and in fact happens more frequently since the period is longer for network updates.
f) You should most probably be using the Projectile class, not FlyingVehicle class!
g) Possibly, but you are correct, shouldn't be affecting you.
h) Aye, very true if you want hundreds of projectiles that are all related/grouped within a cluster.
I'm not sure what you are defining latency as--but it still takes time to pack up the updates, and unpack them on the other side. I'm guessing that the 45-70 is part of that processing time, and not "network ping time" as you are probably defining latency.
I'm reasonably sure that Torque may over-pack, using whatever bytes are necessary to finish a particular packUpdate(), but then stop packing new objects into that particular packet.
#24
Thanks for the clarifications. Except Im not using flyingvehicles as projectiles. Im using flyingvehicles for asteroids (there are aboput 10 in scene flying from here to there). Im using standar projectiles for weapons.
As for the latency, packing and unpacking 200 bytes of data each 200msecs is hardly a processor task that can take 45-70 msecs. But of course my processor can be busy computing the server and client collisions and movement and rendering all at the same time so I guess its completely possible.
Now in the same thread... If I just forget about projectiles... I see my network is still maxed out receiving 215bytes each second. In my simulation I get about 50 bytes transmitted for each flying vehicle each update (I have done testing on this but maybe someone may confirm this amount?)
a. Does it mean that in case I have 4 flying vehicles in scene in players view (thus requiring updates), I will get a saturated network and will get jumps in the simulation because packets are not sent in time?
b. May anyone point me to a resource where I can read about network scoping and get an implementation example? That way I can limit the update of my asteroids to the area I can see thus reducing my network transmissions.
Thank you very much for you patience!
Guimo
12/25/2006 (9:01 am)
StephenThanks for the clarifications. Except Im not using flyingvehicles as projectiles. Im using flyingvehicles for asteroids (there are aboput 10 in scene flying from here to there). Im using standar projectiles for weapons.
As for the latency, packing and unpacking 200 bytes of data each 200msecs is hardly a processor task that can take 45-70 msecs. But of course my processor can be busy computing the server and client collisions and movement and rendering all at the same time so I guess its completely possible.
Now in the same thread... If I just forget about projectiles... I see my network is still maxed out receiving 215bytes each second. In my simulation I get about 50 bytes transmitted for each flying vehicle each update (I have done testing on this but maybe someone may confirm this amount?)
a. Does it mean that in case I have 4 flying vehicles in scene in players view (thus requiring updates), I will get a saturated network and will get jumps in the simulation because packets are not sent in time?
b. May anyone point me to a resource where I can read about network scoping and get an implementation example? That way I can limit the update of my asteroids to the area I can see thus reducing my network transmissions.
Thank you very much for you patience!
Guimo
#25
b) You don't have to implement anything to get pretty decent scoping. You want to prioritize visible stuff higher but not descope objects that are not currently in view - the client/server views may not synch up 100% on every rendered frame so doing this would result in popping.
Finally, and I can't stress this enough, don't abuse objects for things other than their intended use and expect them to work well. The FlyingVehicle class certainly works well for an asteroid but it's also going to be way overkill in terms of both CPU (because it carries a whole lot of extra junk that's needed to have a well-behaving plane, but that's useless or worse for a static object in zero-g), and in terms of networking (because the network code for it is geared towards making a good play experience with people mounted on it, flying past it, etc.).
I would strongly recommend making a new Asteroid class that implements what you need (which is probably spherical collisions, a networking model which attempts to minimize updates, and aggressive optimization for the scenarios where you have a hundred asteroids bumping around in the distance). You might start by taking FlyingVehicle, copying it, then ripping out as much as you can and overriding as much unneeded parent class functionality as possible.
In terms of classes to use, FlyingVehicle and its siblings (WheeledVehicle, HoverVehicle, etc) are about the heaviest weight things in the whole engine, as they derive from nearly every other heavy weight class in the engine. You'd be much better off deriving from GameBase and going from there.
I don't mean to come off as hostile or negative here - I've just seen enough promising games get hamstrung by a refusal or an inability to properly address their core gameplay/performance issues by writing the needed code that I'd hate to have it happen again due to lack of clarity! :)
I'd suggest reading over TNL if you want a better grasp of how Torque's networking works. It's much cleaner, and although many details are different, the overall architecture is identical. www.opentnl.org
You might also want to dredge up the Tribes 1 networking paper and read that.
Finally, look through NetConnection and GameConnection to see how scoping et al is done. netGhost.cc is your primary place of interest.
12/25/2006 (3:49 pm)
A) They require updates when their mask bits are set, not when they're in (or not in) view. If you know that they're following a fixed path you can reduce network traffic tremendously by not sending unneeded updates. Objects which have been skipped in previous updates will get higher priority in the next update, guaranteeing that all things get roughly evenly frequent update rates.b) You don't have to implement anything to get pretty decent scoping. You want to prioritize visible stuff higher but not descope objects that are not currently in view - the client/server views may not synch up 100% on every rendered frame so doing this would result in popping.
Finally, and I can't stress this enough, don't abuse objects for things other than their intended use and expect them to work well. The FlyingVehicle class certainly works well for an asteroid but it's also going to be way overkill in terms of both CPU (because it carries a whole lot of extra junk that's needed to have a well-behaving plane, but that's useless or worse for a static object in zero-g), and in terms of networking (because the network code for it is geared towards making a good play experience with people mounted on it, flying past it, etc.).
I would strongly recommend making a new Asteroid class that implements what you need (which is probably spherical collisions, a networking model which attempts to minimize updates, and aggressive optimization for the scenarios where you have a hundred asteroids bumping around in the distance). You might start by taking FlyingVehicle, copying it, then ripping out as much as you can and overriding as much unneeded parent class functionality as possible.
In terms of classes to use, FlyingVehicle and its siblings (WheeledVehicle, HoverVehicle, etc) are about the heaviest weight things in the whole engine, as they derive from nearly every other heavy weight class in the engine. You'd be much better off deriving from GameBase and going from there.
I don't mean to come off as hostile or negative here - I've just seen enough promising games get hamstrung by a refusal or an inability to properly address their core gameplay/performance issues by writing the needed code that I'd hate to have it happen again due to lack of clarity! :)
I'd suggest reading over TNL if you want a better grasp of how Torque's networking works. It's much cleaner, and although many details are different, the overall architecture is identical. www.opentnl.org
You might also want to dredge up the Tribes 1 networking paper and read that.
Finally, look through NetConnection and GameConnection to see how scoping et al is done. netGhost.cc is your primary place of interest.
#26
I dont think your message was hostile at all, unfortunately truth has always a sharp edge.
I completely agree thet using a FlyingVehicle object is an overkill for an steroid and creating a new class in order to handle an asteroid is completely appropiate... but I was really hoping to shorten my development time by using an existing class.
I started using an Item class for my asteroids but projectiles didnt collide with them. So I decided using the flying vehicle as it was able to use particles, explosions, collisions and all.
Anyway now I need to create two classes. An Asteroid class and a ProjectileCluster class... I really hope thats all... :)
Thank you very much for your help!
Luck!
Guimo
P.D. I really should make this an step by step tutorial for beginners like me. How to build a game from ground up. Would anybody review it?
12/25/2006 (8:15 pm)
Hi Ben,I dont think your message was hostile at all, unfortunately truth has always a sharp edge.
I completely agree thet using a FlyingVehicle object is an overkill for an steroid and creating a new class in order to handle an asteroid is completely appropiate... but I was really hoping to shorten my development time by using an existing class.
I started using an Item class for my asteroids but projectiles didnt collide with them. So I decided using the flying vehicle as it was able to use particles, explosions, collisions and all.
Anyway now I need to create two classes. An Asteroid class and a ProjectileCluster class... I really hope thats all... :)
Thank you very much for your help!
Luck!
Guimo
P.D. I really should make this an step by step tutorial for beginners like me. How to build a game from ground up. Would anybody review it?
#27
By following your suggestions I duplicated the Vehicle class and the FlyingVehicle class into a SimpleVehicle class and Assteroid class respectively. Then I stripped a lot of code from them (I really doubt asteroids will splash into water or use hover jets anytime).
By removing the transmission of the angular momentum and velocity and the steering information plus some other attributes on the update code, I managed to reduce the transmitted bytes from 54 bytes for a FlyingVehicle to 24 bytes for my asteroid class. That allows me to keep collisions and physical simulations while transmitting data for 7-8 asteroids. Gameplay is more fluid now and I get no network stutter. This has helped me whith the original weapon firing problem but the problem is still there. Im still working on the ProjectileCluster class in order to solve this.
Thank you very much, this has been very interesting.
Luck!
Guimo
12/27/2006 (7:26 am)
Hi all!By following your suggestions I duplicated the Vehicle class and the FlyingVehicle class into a SimpleVehicle class and Assteroid class respectively. Then I stripped a lot of code from them (I really doubt asteroids will splash into water or use hover jets anytime).
By removing the transmission of the angular momentum and velocity and the steering information plus some other attributes on the update code, I managed to reduce the transmitted bytes from 54 bytes for a FlyingVehicle to 24 bytes for my asteroid class. That allows me to keep collisions and physical simulations while transmitting data for 7-8 asteroids. Gameplay is more fluid now and I get no network stutter. This has helped me whith the original weapon firing problem but the problem is still there. Im still working on the ProjectileCluster class in order to solve this.
Thank you very much, this has been very interesting.
Luck!
Guimo
#28
Glad to see this is working for you, thanks for taking the time to set things up such that you can properly solve your problems! :)
12/27/2006 (10:39 am)
Guimo - you might be able to significantly reduce the number of updates sent for your asteroid class. I believe vehicles like to set their dirty bits every tick; however, an asteroid is likely to drift in an entirely deterministic way for some time before needing an update. If you can reliably detect when an update might be needed (e.g. only set dirty bits when an impulse or collision has occured) then you might be able to go from 600 updates a minute per asteroid to many, many fewer.Glad to see this is working for you, thanks for taking the time to set things up such that you can properly solve your problems! :)
Torque 3D Owner Guimo
In fact I've just found an excellent tool to help tracking everything. Look at this in your default.bind.cs:
moveMap.bindCmd(keyboard, "v", "NetGraph::toggleNetGraph();", "");
This will popup a frame detailing the number of objects in the simulation, bytes sent, bytes read, updates, latency and packets lost. Really helpful.
For me the information reads (when not firing!)
Objects 30
Bytes Sent 200
Latency 45 - 70
Ghost updates 3-4
Bytes Received 215 (looks like Im maxed out)
Packet lost 0
Now when I remove the asteroids (which are flyingvehicles) the data read:
Objects 20 (10 asteroids not present)
Bytes Sent 200
Latency 45 - 70
Ghost updates 1
Bytes Received 49 (no longer maxed!)
Packet lost 0
Removing the 20 powerups (Item objects) in my game does not have any effect in the network. So it looks like there is something strange with using the FlyingVehicles as objects or maybe Ive broke the networking when creating my new attributes... I'm checking this out!
@Flybynight
I know I still have a lot of research to do. But whay I have understood from the posts:
a. Torque sends 200 bytes of information each update.
b. Each time an object is created in the server, a ghost is sent to the clients. This uses a part of the 200 bytes limit.
c. Any event will be sent to the clients and will take some of the 200 bytes limit
d. Projectiles are sent but they have no priority. They will be sent if bytes are left.
e. Any data not sent is queued and will be sent in the next tick (32msecs later). This will cause the queue to max out and eventually packets out of time wont be sent. Of course projectiles which have a little lifetime will be destroyed before being sent.
f. I may have broken the networking code and Im sending too much data each frame even when not firing. I need to find where.
d. Torquescript may cause an slight overhead but not significant for a simple game like mine.
g. Creating a projectile cluster object will help me a lot in my game because a single ghost will control a lot of objects so reducing the networking transmissions.
What bothers me and just dont get it... if Im running in a single machine and hosting no game (single player).
Why do I still have latency? Latency would mean Im still sending the data through a physical network. I understand the networking layer will solve any transmission protocol/delay/retransmission. But if Im not physically connected to a network... what causes the latency? Is it caused by the CPU speed? Thats unlikely. Network hardware is not an issue because that would mean Torque requires a network card to work which is a condition I really doubt.
So the only limitation would be that some value limits the transmission to 200 bits/second thus causing the latency...does moving up/down this value will affect the simulation? (Im not telling this is the solution, Im just wondering about this).
Thank you very much for your responses.
Luck!
Guimo