Memory Leak / Performance Slowdown?
by Chris Jorgensen · in Torque Game Builder · 08/30/2006 (12:45 pm) · 18 replies
Hi all,
I'm working on a little 2-player 2D shooter in TGB ala Star Control Melee. For some reason, after the game is running for a while, it starts to get choppy and slow down. I'm trying to figure out the cause of this. I'm running on a Dell Inspiron 2200 -- not the most powerful computer in the world, but I figure it should be able to handle a 2D game. Here are the things I *think* might be contributing to the problem, but I don't know:
1- There are a lot of sprites. Maybe 100 stars (using sprites to make their positions random), and a dozen gas clouds ranging in sizes from 240x240 to 1600x800 or so, two ships (62x46 big). Or maybe the bigger sprites are too big?
2- Things are scheduled often. I have things like real-time camera following and zoom. (Though I think I'll be changing the zoom soon.) So the schedule(time, something, something) gets called a lot. I used this same script technique for some simple placeholder AI as well.
3- Sloppy implentation of a script. I have nothing in mind, but is it possible to write a bit of script so inefficiently that its ill-effects build up?
Anyone have any ideas?
I'm working on a little 2-player 2D shooter in TGB ala Star Control Melee. For some reason, after the game is running for a while, it starts to get choppy and slow down. I'm trying to figure out the cause of this. I'm running on a Dell Inspiron 2200 -- not the most powerful computer in the world, but I figure it should be able to handle a 2D game. Here are the things I *think* might be contributing to the problem, but I don't know:
1- There are a lot of sprites. Maybe 100 stars (using sprites to make their positions random), and a dozen gas clouds ranging in sizes from 240x240 to 1600x800 or so, two ships (62x46 big). Or maybe the bigger sprites are too big?
2- Things are scheduled often. I have things like real-time camera following and zoom. (Though I think I'll be changing the zoom soon.) So the schedule(time, something, something) gets called a lot. I used this same script technique for some simple placeholder AI as well.
3- Sloppy implentation of a script. I have nothing in mind, but is it possible to write a bit of script so inefficiently that its ill-effects build up?
Anyone have any ideas?
About the author
Owner of Cascadia Games LLC
#2
I know I've seen it pretty much drag my project to a halt..
I wind up doing stuff like:
if ($someglobal) echo("my stuff");
This way I can turn the echoed commands on/off..
Just an Idea.
08/30/2006 (12:53 pm)
If you have a lot of echo("") commands for debugging purposes, especially in timer callbacks, or scheduled calls, this could be the cause.I know I've seen it pretty much drag my project to a halt..
I wind up doing stuff like:
if ($someglobal) echo("my stuff");
This way I can turn the echoed commands on/off..
Just an Idea.
#3
That onUpdateScene suggestion is another good idea. I do have a lot of schedule's floating around and had suspected they might have been an issue.
Thanks to both the ideas. :)
08/30/2006 (3:46 pm)
^Ooh, echo's is a good point just for performance in general. Reminds me of pulling printk's out of linux drivers to improve performance...That onUpdateScene suggestion is another good idea. I do have a lot of schedule's floating around and had suspected they might have been an issue.
Thanks to both the ideas. :)
#4
08/31/2006 (11:41 am)
Good point Scott. In a tutorial I did back in the day (Object Selection Part 3) I went over encompassing your echo's in statements like that and filtering it to different systems other than just 'echo'. echo and error are great for quick tests, though be sure to remove them otherwise your console gets flooded and that eats up memory fairly quickly... if you flood it too much it can cause a crash as well (or massive console.log writing when exiting).
#5
Then run your game for a minute or two, then type this in the console.
and you should see a lot of info telling you what is using most of the processing time.
08/31/2006 (11:44 am)
Also you can use the profiler system... type this in the console when running your game.profilerEnable(true);
Then run your game for a minute or two, then type this in the console.
profilerDump();
and you should see a lot of info telling you what is using most of the processing time.
#6
Update: actually, I just found a bunch of schedules in the weapon classes of all places. I've clipped those out and the game seems to be better. But I haven't thoroughly verified I don't have the slowdown problem any more.
09/02/2006 (5:57 pm)
^Interesting. SimAdvanceTime seems oddly large, starting at 7% when the game starts to %16 when it starts getting choppy. It's just a stab in the dark, but my guess is that has to do with the "schedule" calls. I tried merging all my scheduled events into 2 blocks (ai and screen events (like camera move/zoom)) but I haven't seen any notable speedup. Hmmm... well, more experimenting to do!Update: actually, I just found a bunch of schedules in the weapon classes of all places. I've clipped those out and the game seems to be better. But I haven't thoroughly verified I don't have the slowdown problem any more.
#7
Edit: nevermind, the answer is yes. I found that here: http://www.garagegames.com/mg/forums/result.thread.php?qt=42075
09/05/2006 (10:06 am)
Okay, so I tried merging the schedule's and that's helped a little bit. There's still some slowdown though. Would switching from sprites to tiles for background objects affect performance significantly?Edit: nevermind, the answer is yes. I found that here: http://www.garagegames.com/mg/forums/result.thread.php?qt=42075
#8
09/08/2006 (4:07 pm)
This is just an update for anyone curious. I switched from a global scheduler to onUpdateScene for my camera move/zoom control. I'd say there's been a pretty notable improvement in performance. I've also clipped about half of the schedule calls out. So that definitely was the problem! 
#9
Probably best to grab a screenie of the debug-banner when the slowdown is happening.
- Melv.
09/09/2006 (4:57 am)
You could also turn-on the debug-banner (and read the associated doco) to see if there's something very odd going on there. For instance, you might find that the object count is suddenly growing or that there are lots of bin-relocations etc.Probably best to grab a screenie of the debug-banner when the slowdown is happening.
- Melv.
#10
So the game is running much better now. There's really only 1 scenario that isn't super smooth now. Basically, if the camera stays fixed on all the ships (2 for the moment, eventually will be 4 I hope). The camera zooms in when they're close and zooms out when they're far apart. The very first zoom-out always has a little bit of delay before it happens. Then they're smooth from then on. I don't know if it's just the processor caching the art or what, but it's consistently like that.
09/25/2006 (12:06 pm)
I finally pushed over the AI to the onUpdateScene function instead of using schedule calls over and over. Suprisingly, the slowdown dramatically increased! Upon further inspection I realized that, in fact, there was some bad scripting that caused the slowdown. The AI FSM, from what I could tell, had some state-thrashing occurring. I fixed the state-locking (that is, the AI stays in one state only) and the AI-induced slowdown has stopped. So the game is running much better now. There's really only 1 scenario that isn't super smooth now. Basically, if the camera stays fixed on all the ships (2 for the moment, eventually will be 4 I hope). The camera zooms in when they're close and zooms out when they're far apart. The very first zoom-out always has a little bit of delay before it happens. Then they're smooth from then on. I don't know if it's just the processor caching the art or what, but it's consistently like that.
#11
09/25/2006 (1:37 pm)
If schedulers are causing increasing performance hits the further you go in the game. Could it be that something is not getting destroyed either in script or c++ code. Ie like an array of schedule events that keeps getting appended to so that it gets increasingly larger as time goes on. Because if that is the case, then reducing the number of schedules will help initially but the overall problem will still come back after a while of playing.
#12
Say for example you had a recurring schedule to occur every 100 milliseconds to tick your AI Manager. Now, think about the frequency of ticks that putting the ticker into ::onUpdateScene() would have given say a frame rate of 100 FPS--that's 100 ticks per second, or the same as scheduling the update every millisecond.
I've posted this in a couple of the other threads in the past, but to reinforce:
scheduling an event does two things:
--creates a new internal C++ object called SimEvent, and gives it some data (specifically, the function you want scheduled, plus the arguments to the schedule).
--places the SimEvent on the object processing queue at currentSimTime+ (millisecond delay /millisecondsPerSimTick), where millisecondsPerSimTick isn't an actual variable, but a concept based on some internal constants. It's roughly 33 milliseconds assuming TGB hasn't modified it.
Once the processing queue matures to the posted tick for the SimEvent, the SimEvent is processed, which in turn will call your function you originally scheduled. Once the function returns, the SimEvent is destroyed.
As an aside, this is also why schedules aren't guaranteed to be exact in real time--once the event maturation SimTick interval is calculated, the sim waits for that simTime to occur, and is disconnected from realTime. Stock SimEvents don't monitor anything like time passed since event posting, but with a Pro license if this is something your game absolutely must have, you could implement something to that effect.
Minor Edit: changed misleading "time" word to more appropriate "tick" in the next to last paragraph.
09/25/2006 (3:33 pm)
As a general rule, properly handled schedules will be orders of magnitude more efficient than putting a ticker into ::onUpdateScene().Say for example you had a recurring schedule to occur every 100 milliseconds to tick your AI Manager. Now, think about the frequency of ticks that putting the ticker into ::onUpdateScene() would have given say a frame rate of 100 FPS--that's 100 ticks per second, or the same as scheduling the update every millisecond.
I've posted this in a couple of the other threads in the past, but to reinforce:
scheduling an event does two things:
--creates a new internal C++ object called SimEvent, and gives it some data (specifically, the function you want scheduled, plus the arguments to the schedule).
--places the SimEvent on the object processing queue at currentSimTime+ (millisecond delay /millisecondsPerSimTick), where millisecondsPerSimTick isn't an actual variable, but a concept based on some internal constants. It's roughly 33 milliseconds assuming TGB hasn't modified it.
Once the processing queue matures to the posted tick for the SimEvent, the SimEvent is processed, which in turn will call your function you originally scheduled. Once the function returns, the SimEvent is destroyed.
As an aside, this is also why schedules aren't guaranteed to be exact in real time--once the event maturation SimTick interval is calculated, the sim waits for that simTime to occur, and is disconnected from realTime. Stock SimEvents don't monitor anything like time passed since event posting, but with a Pro license if this is something your game absolutely must have, you could implement something to that effect.
Minor Edit: changed misleading "time" word to more appropriate "tick" in the next to last paragraph.
#13
Well, cool. It'll be nice to get this problem out of the way. :)
09/25/2006 (5:39 pm)
Interesting. I don't think an AI manager has strict real-time requirements. I might have to try out "schedule" again for the AI, now that I've found the problem wasn't related to that function. In fact, like I said, things ran better with the schedule calls in place. That would leave only my camera movement/zoom in the onUpdateScene call.Well, cool. It'll be nice to get this problem out of the way. :)
#14
Just a note, ::onUpdateScene at 100fps is actually called every 10 milliseconds, not every millisecond ;)
09/25/2006 (6:16 pm)
@StephenJust a note, ::onUpdateScene at 100fps is actually called every 10 milliseconds, not every millisecond ;)
#15
09/25/2006 (9:52 pm)
Good point, my math was off, hehe.
#16
Is this possible trough non-pro script ?
09/28/2006 (7:20 am)
So if I want to have real "real-time" schedule (so it wont be dependant on FPS or whatever), what must I do ? Is this possible trough non-pro script ?
#17
09/28/2006 (9:14 am)
To get "realtime" results (which is a word loaded with so many possible meanings) you can use schedule, timer, or onUpdateScene methods. The key is to calculate the time since the last call of the method and scale all calculations based on the actual time passed. Basically, never assume the same amount of time has passed between each call.
#18
My laptop has an integrated Intel graphics device. Someone told me that non-power-of-2 sized images can cause slowdown on a device like this. Most of the nebula images are not power-of-2 sized. Is this the most likely issue here?
I appreciate the feedback btw. The game is really coming along nicely. I wish I could spend 8 hours a day working on it, rather than diagnostic/driver code all day!
UPDATE: Arg! Okay, yes, the level did speed up with power-of-two sizes... but even on a blank level, if the AIs battle for a while, the whole game slows down to a choppy speed. If I end the level and reload it, things run fast again. Dang, what an annoying problem!
UPDATE TWO: Well, so it looks like I figured out the problem. If you've got, say, a homing missile that uses schedule calls to keep it tracking, and the missile hits its target, the sprite gets deleted but the schedule loop keeps going unless you give it a flag like "if missile alive then schedule." This same thing was happening with the AI. So that appears to have helped a bit as well.
09/29/2006 (11:37 am)
Okay, I spoke too soon. I *still* get slowdown but it appears to be of a different nature now. I have two test levels. Both have 200 random stars or so (sprites for now but that might change). One has just a planet in the background (512 by 512). It runs perfectly smooth. The other has about 5 large (over 1024x1024) nebula images in the background, and then some "fog" sprites to help give the level a little depth. That level runs extremely slow.My laptop has an integrated Intel graphics device. Someone told me that non-power-of-2 sized images can cause slowdown on a device like this. Most of the nebula images are not power-of-2 sized. Is this the most likely issue here?
I appreciate the feedback btw. The game is really coming along nicely. I wish I could spend 8 hours a day working on it, rather than diagnostic/driver code all day!
UPDATE: Arg! Okay, yes, the level did speed up with power-of-two sizes... but even on a blank level, if the AIs battle for a while, the whole game slows down to a choppy speed. If I end the level and reload it, things run fast again. Dang, what an annoying problem!
UPDATE TWO: Well, so it looks like I figured out the problem. If you've got, say, a homing missile that uses schedule calls to keep it tracking, and the missile hits its target, the sprite gets deleted but the schedule loop keeps going unless you give it a flag like "if missile alive then schedule." This same thing was happening with the AI. So that appears to have helped a bit as well.
Torque 3D Owner Matthew Langley
Torque