Plan for Jon Frisby
by Jon Frisby · 10/18/2005 (10:47 am) · 11 comments
Background
So here I am, working on a game in T2D. Unfortunately timing and audio really really matter for my game, and these seem to be two area where T2D is... incomplete, at best. Think of it as being like a rhythm game (Dance Dance Revolution) -- movement on screen needs to be synchronized to the music (as do sound effects based on user action). I.E. object X must collide with object Y on a 1/16th note boundary (give or take a couple milliseconds).
Which brings us to the heart of the matter: Sim-time versus Real-time.
The music plays in accordance with real-time, which is proper and correct. Physics and scheduled events happen in accordance with sim-time. What is the relationship between the two?
Heisenberg Strikes Again
I set up a scheduled event to measure the rate of change in sim-time and real-time, and discovered that they move along at a more-or-less similar rate. There is no obvious multiplier or geometric difference, which makes things a little easier (yes, theres $timeScale, but I'm not letting users muck about with that...). HOWEVER, the kicker is that they're not quite exactly synchronized. Sometimes sim-time is a little faster than real-time. Sometimes it's a little slower. In my tests on a 1.33Ghz G4, T2D 1.0.2, in my game, the variance was 960-1,024 ms of real-time for 1,000 ms of sim-time.
The kicker is that the more precisely I try to measure it, the greater the disparity. If I schedule my routine to run every 10 ms , the difference is at LEAST +/- 4. So just like uncertainty, measuring a thing affects it (yes yes yes, it's not QUITE the same, but I like the analogy, dangit!).
Combing through the source code, and dusting off the cobwebs (haven't touched C++ since before it had an ANSI/ISO spec), it looks like the progression of time is affected by the amount of time the engine spends doing stuff. Whether that's rendering, applying physics, or running script code. The more the engine has to do, the less likely it's going to track time accurately.
Relativity to the Rescue?!
So what to do then? My current thinking is to have a period scheduled event that determines the delta since the last time, and compensates. In the case of my game, I have one moving object whose *correct* position could be determined pretty easily mathematically based on how long the music's been playing, and what the tempo is. My job could determine if the object was ahead, or behind of where it needed to be, and make a subtle adjustment to its velocity to cause it to catch up to (or wait for) the music. The problem is I want the adjustments to be as tiny as possible to minimize the chance of the user *noticing* them. But as before, the more frequently I measure, the larger the discrepency. I could easily wind up oscilating. I have this nasty nagging feeling that implementing something like this will involve relativity, or at the VERY least diff-eq. Blech.
Unfortunately, I'm not sure how this will behave in a low-frame-rate environment. I've already seen some bizarre weirdness that I just can't grok when the framerate drops too far, even for a split second.
Please, please, PLEASE won't somebody tell me I'm going down the wrong path and that there's a simple and Correct way to do this that doesn't involve such an ugly hack?!
Oh yeah, obligatory screen shot:
So here I am, working on a game in T2D. Unfortunately timing and audio really really matter for my game, and these seem to be two area where T2D is... incomplete, at best. Think of it as being like a rhythm game (Dance Dance Revolution) -- movement on screen needs to be synchronized to the music (as do sound effects based on user action). I.E. object X must collide with object Y on a 1/16th note boundary (give or take a couple milliseconds).
Which brings us to the heart of the matter: Sim-time versus Real-time.
The music plays in accordance with real-time, which is proper and correct. Physics and scheduled events happen in accordance with sim-time. What is the relationship between the two?
Heisenberg Strikes Again
I set up a scheduled event to measure the rate of change in sim-time and real-time, and discovered that they move along at a more-or-less similar rate. There is no obvious multiplier or geometric difference, which makes things a little easier (yes, theres $timeScale, but I'm not letting users muck about with that...). HOWEVER, the kicker is that they're not quite exactly synchronized. Sometimes sim-time is a little faster than real-time. Sometimes it's a little slower. In my tests on a 1.33Ghz G4, T2D 1.0.2, in my game, the variance was 960-1,024 ms of real-time for 1,000 ms of sim-time.
The kicker is that the more precisely I try to measure it, the greater the disparity. If I schedule my routine to run every 10 ms , the difference is at LEAST +/- 4. So just like uncertainty, measuring a thing affects it (yes yes yes, it's not QUITE the same, but I like the analogy, dangit!).
Combing through the source code, and dusting off the cobwebs (haven't touched C++ since before it had an ANSI/ISO spec), it looks like the progression of time is affected by the amount of time the engine spends doing stuff. Whether that's rendering, applying physics, or running script code. The more the engine has to do, the less likely it's going to track time accurately.
Relativity to the Rescue?!
So what to do then? My current thinking is to have a period scheduled event that determines the delta since the last time, and compensates. In the case of my game, I have one moving object whose *correct* position could be determined pretty easily mathematically based on how long the music's been playing, and what the tempo is. My job could determine if the object was ahead, or behind of where it needed to be, and make a subtle adjustment to its velocity to cause it to catch up to (or wait for) the music. The problem is I want the adjustments to be as tiny as possible to minimize the chance of the user *noticing* them. But as before, the more frequently I measure, the larger the discrepency. I could easily wind up oscilating. I have this nasty nagging feeling that implementing something like this will involve relativity, or at the VERY least diff-eq. Blech.
Unfortunately, I'm not sure how this will behave in a low-frame-rate environment. I've already seen some bizarre weirdness that I just can't grok when the framerate drops too far, even for a split second.
Please, please, PLEASE won't somebody tell me I'm going down the wrong path and that there's a simple and Correct way to do this that doesn't involve such an ugly hack?!
Oh yeah, obligatory screen shot:
#2
10/18/2005 (10:53 am)
Think "Lumines" (PSP). I have some gameplay variations I intend to try, but I'm definitely going for the same fusion of visual artistry, aural artistry, and gameplay. 
#3
A) The entire idea of tying real time to "sim time" (not just Torque related) to "real time" is an incredibly difficult challenge, and one that has been addressed (not always successfully) a ton of times in the past. Ada (the "original" Department of Defense language) is a language designed to allow for the concept of "real time/embedded programming", which in a nutshell hands everything in the call stack a "real time slice" of duration in which the execution of code within that particular stack frame must execute. This challenge hits in so many areas that it is amazing: from true embedded systems (flight control algorithms in the F-16 for example, the "fly by wire" system), to instrumentation based simulations (where you have a ton of environmental reporting mechanisms that communicate real world environment changes to your simulation) i.e. something like a control systems overlay to a plant or assembly line or similar concepts, to the type of thing where you are describing where simulation advancement must be tied very directly to fixed duration/speed events such as playing media files.
Torque's simulation wasn't designed to address real time sychronization in any real sense, so it's a challenge to do something similar to what you are talking about out of the box.
Now, some of the things you can consider:
--one of the challenges of any software development is to describe in the most accurate and minute detail possible the requirements of your project, without dictating implementation strategies or requirements. For example, instead of saying "The project will use a FIFO queue abstract data type and the Torque event scheduling system to simulate a cook adding dishes to be cleaned by the dishwasher", you would want to re-phrase it to something like "The dishwasher will clean every dish as it become available to them, and dirty dishes will be made available to the dishwasher as they are delivered".
Sounds trivial/pedantic, but it can actually be critical in how your project evolves in both design and implementation. You may want to try creating detailed descriptions of your requirements that are implementation agnostic, and see if anything jumps out at you.
Related to this topic: you want to design at the most minute detail level possible, and then regress the design/algorithm in use to the most simplistic possible to accomplish the original design requirements. Again, sounds simple, but think about it from the perspective of modelling -any- real world physics:
While we have equations that describe the physics of the universe as we know it, those equations are actually more simplistic representations of the underlying (quantum) mechanics of what is actually going on. Instead of modelling quantum theory for example (our best known "design requirements"), we model and implement a reduced version of simplified equations of physics that can approximate quantum mechanics. In other words, we describe with as most detail as possible, and then implement with the least amount of detail/accuracy possible to meet our goals.
How does this relate to your issue?
Well, one of the things that the Theora plug-in provides is a method for synchronizing the audio track to the video track (NOTE: my info here is from a basic conversation with Ben Garney as he was writing the theora plug-in, so is almost certain to not be quite accurate!). While it -appears- that the video is playing completely in parallel with the audio (or in our minds even the exact same "track"), it's actually going through a synchronization process that slows and speeds up each of the two medias to maintain synchronization.
You could use something similar to this idea with your issue: read ahead in the audio stream, identify and isolate a "1/16th note event", and then schedule a high prority simulation pulse to occur that is synchronized with your audio track.
Alternatively, control the speed of playback of your audio track via feedback mechanisms between that playback and the simulation itself so that the simulation can fine-tune the speed of the audio playback to more accurately coincide with the simulation events. Of course you want to be very careful to not cause distortion, etc., but it's a matter of fine tuning I think within the parameters of human tolerance.
Long story short: Yes, your situation is complicated, and that is compounded by the fact that Torque, C++, and the underlying operating systems weren't designed with real time synchronization in mind. You are definiately on the right track, but instead of worrying about Relativity and quantum theory in your implementation, come up with approximate implementations that approach those models, without requiring 1,000 networked Cray supercomputers ;)
10/18/2005 (12:12 pm)
A couple of interesting ideas/points from my perspective (unrelated really to GG's perspective!):A) The entire idea of tying real time to "sim time" (not just Torque related) to "real time" is an incredibly difficult challenge, and one that has been addressed (not always successfully) a ton of times in the past. Ada (the "original" Department of Defense language) is a language designed to allow for the concept of "real time/embedded programming", which in a nutshell hands everything in the call stack a "real time slice" of duration in which the execution of code within that particular stack frame must execute. This challenge hits in so many areas that it is amazing: from true embedded systems (flight control algorithms in the F-16 for example, the "fly by wire" system), to instrumentation based simulations (where you have a ton of environmental reporting mechanisms that communicate real world environment changes to your simulation) i.e. something like a control systems overlay to a plant or assembly line or similar concepts, to the type of thing where you are describing where simulation advancement must be tied very directly to fixed duration/speed events such as playing media files.
Torque's simulation wasn't designed to address real time sychronization in any real sense, so it's a challenge to do something similar to what you are talking about out of the box.
Now, some of the things you can consider:
--one of the challenges of any software development is to describe in the most accurate and minute detail possible the requirements of your project, without dictating implementation strategies or requirements. For example, instead of saying "The project will use a FIFO queue abstract data type and the Torque event scheduling system to simulate a cook adding dishes to be cleaned by the dishwasher", you would want to re-phrase it to something like "The dishwasher will clean every dish as it become available to them, and dirty dishes will be made available to the dishwasher as they are delivered".
Sounds trivial/pedantic, but it can actually be critical in how your project evolves in both design and implementation. You may want to try creating detailed descriptions of your requirements that are implementation agnostic, and see if anything jumps out at you.
Related to this topic: you want to design at the most minute detail level possible, and then regress the design/algorithm in use to the most simplistic possible to accomplish the original design requirements. Again, sounds simple, but think about it from the perspective of modelling -any- real world physics:
While we have equations that describe the physics of the universe as we know it, those equations are actually more simplistic representations of the underlying (quantum) mechanics of what is actually going on. Instead of modelling quantum theory for example (our best known "design requirements"), we model and implement a reduced version of simplified equations of physics that can approximate quantum mechanics. In other words, we describe with as most detail as possible, and then implement with the least amount of detail/accuracy possible to meet our goals.
How does this relate to your issue?
Well, one of the things that the Theora plug-in provides is a method for synchronizing the audio track to the video track (NOTE: my info here is from a basic conversation with Ben Garney as he was writing the theora plug-in, so is almost certain to not be quite accurate!). While it -appears- that the video is playing completely in parallel with the audio (or in our minds even the exact same "track"), it's actually going through a synchronization process that slows and speeds up each of the two medias to maintain synchronization.
You could use something similar to this idea with your issue: read ahead in the audio stream, identify and isolate a "1/16th note event", and then schedule a high prority simulation pulse to occur that is synchronized with your audio track.
Alternatively, control the speed of playback of your audio track via feedback mechanisms between that playback and the simulation itself so that the simulation can fine-tune the speed of the audio playback to more accurately coincide with the simulation events. Of course you want to be very careful to not cause distortion, etc., but it's a matter of fine tuning I think within the parameters of human tolerance.
Long story short: Yes, your situation is complicated, and that is compounded by the fact that Torque, C++, and the underlying operating systems weren't designed with real time synchronization in mind. You are definiately on the right track, but instead of worrying about Relativity and quantum theory in your implementation, come up with approximate implementations that approach those models, without requiring 1,000 networked Cray supercomputers ;)
#4
10/18/2005 (12:13 pm)
If you're interested in tightly synchronizing to music, you might want to browse through the Stepmania source code sometime(it is being used by several commercial music game projects now) and take a look at how they are doing timing. What I remember reading about it is that they have some pretty low-level communication with the sound card going on so that gameplay timings don't get affected by the OS or drift. However, it still screws up in the worst cases when the hard drive starts churning and cpu is pegged at 100%.
#5
Having already specified the requirements, my issue becomes implementing a solution. After considering and discarding various ideas, the one I proposed was the one I came up with. I posted here looking for somewhat more concrete advice in the off-chance I was fundamentally misunderstanding Torque, but your comments imply that I am not.
The notion of synchronizing the music to sim-time (what you're essentially suggesting with the comment about the Theora plugin) is the most useful part of your comment, but that idea would work only if the deviations were small, and presently the deviations I'm seeing are not terribly small (64ms is pretty huge). For such large deviations, the change in tempo would definitely be heard by the user. It also requires hacking around in C++, and I neglected to mention that I'm trying to minimize that as much as possible...
Letting the problem toss around in my head I've come to the notion that my best bet might be to discard the physics system entirely and just move my object manually in small increments. Doing that efficiently and smoothly would mean coding in C++ though.
@James: Yes, I'd considered looking at StepMania, but was hoping there was a way I could do this without having to gut Torque.
10/18/2005 (2:14 pm)
@Stephen: I appreciate that you're trying to be helpful, but I've been programming for ~14 years, 8 of those professionally. I'm fully aware of the perils of specifying an implementation instead of requirements, and so forth. Not a problem here. I'm long past the requirements specification phase. The requirements are simple: Object X must make contact with object Y at roughly time Z, and sound effects must play in accordance with that. Z is defined in terms of basic music theory -- there's no need to read ahead in the audio stream to figure it out -- it's trivial math. If tempo = 60 beats per minute, and elapsed time since start of music is 6,972 ms, and my timing signature is 4/4, then the next 16th note is 28ms from now (and incidentally my bar should be past the 11th column since 27 16th notes have passed, and there are 16 columns). Hard real-time isn't even remotely needed here -- SOFT real-time isn't even remotely needed to fulfill the requirement. Under normal circumstances I could tolerate about up to about 16ms of latency before the user really noticed anything wrong -- but there are a couple factors that may change that (visual perception is quite keen, so the user might notice much smaller deviances because of the lack of visual synchronization, but user engrossment in gameplay may compensate for that somewhat...).Having already specified the requirements, my issue becomes implementing a solution. After considering and discarding various ideas, the one I proposed was the one I came up with. I posted here looking for somewhat more concrete advice in the off-chance I was fundamentally misunderstanding Torque, but your comments imply that I am not.
The notion of synchronizing the music to sim-time (what you're essentially suggesting with the comment about the Theora plugin) is the most useful part of your comment, but that idea would work only if the deviations were small, and presently the deviations I'm seeing are not terribly small (64ms is pretty huge). For such large deviations, the change in tempo would definitely be heard by the user. It also requires hacking around in C++, and I neglected to mention that I'm trying to minimize that as much as possible...
Letting the problem toss around in my head I've come to the notion that my best bet might be to discard the physics system entirely and just move my object manually in small increments. Doing that efficiently and smoothly would mean coding in C++ though.
@James: Yes, I'd considered looking at StepMania, but was hoping there was a way I could do this without having to gut Torque.
#6
10/18/2005 (2:33 pm)
I left out a logical leap there that I probably ought to note -- I continue to pursue "easy" strategies despite StepMania's need to talk to the video card for the simple reason that the synchronization (or lack thereof) does not impact the user's score, and so doesn't need to be ultra-precise. It's about creating a particular experience for the user. I can tolerate some wackiness if the disk thrashes or something causes things to grind to a halt, provided I can recover and get back in sync reasonably soon thereafter. 
#7
10/19/2005 (3:30 am)
It looks really like Lumines. It's really a great game. Somebody have tried meteos on the ds ? 
#8
Ya, that is nothing like my game :)
Mine is something completely rediculous.
10/19/2005 (3:59 am)
I read up on Lumines...Ya, that is nothing like my game :)
Mine is something completely rediculous.
#9
12/13/2006 (4:16 pm)
Hey Jon, did you ever find a workaround? I could also use a "sim-independent" schedule function since I have to do a reaction test for users and they have to react in "real" time, e.g. 1000ms, not 923 or 956 or something which the sim-time based schedule() gives me.... :(
#11
Care to share some more details though?
12/18/2006 (8:25 am)
Hi Jon, thanks for your answer! Haven't tried ITickable yet, but is it really totally independent of the current engine weight load and the sim time etc.? It just sounds like a way to get objects to receive game ticks... guess I just gotta try it, once that damn cold goes away and my head is able to think again... :PCare to share some more details though?
Torque Owner Chris Labombard
Premium Preferred
It is very very important in the game I am building as well. For a minute I thought you were describing my game... Could you explain your game in a bit more detail. WHat is hte game play? The concept ?