Blog for Clark Fagot at GarageGames.com

Tuesday Jun 14 18:13

Clark Fagot — 2005-06-14T18:13:22+00:00

Thoughts on the BraveTree/GarageGames merger/acquisition.

I'm once again going against Joe's advice and posting a plan with no pretty pictures.

This plan is actually just going to be a re-hash of my comments in this thread regarding the benefits to all (including the community) of the BraveTree and GarageGames merger (note how pride makes me call it a merger not an acquisition).

We've all been excited about this move for a while. It's been a definite positive for our productivity. I no longer have to talk to accountants, lawyers, and insurance agents (although they are all perfectly nice people, just in case any of them are reading this :). Joe no longer has to get on the phone to do biz dev. In fact, I work from home several days a week now (I sit on my couch at 10:30am as I type), simply because I can and I like the change of environment.

But the benefits go well beyond that. To answer some of the questions that Brian Russey (rightly) raises in the thread I linked to above, allow me to focus on how this change affects the work I do.

Any little project I decide to pick up and work on for a week can now be a productive task rather than blue sky work where the payout is uncertain. E.g., at the beginning of the year I spent some evenings allowing Torque to do scalable shells (which means that the screen pretends it's, say, 800x600 even when it's bigger or smaller). Anyone who has worked on creating a multi-res shell knows what a nice feature this is (previously only available in the Unreal engine). It's the type of project that doesn't really work as a resource because there are so many teeny (but important) changes to make. But it can easily be added to a future version of Torque and suddenly everyone has access to it just like that. Before, although I could contribute the code back to Torque (and in the past I had done that with a number of enhancements and bug fixes) it didn't really behoove me to make a practice of doing new things and then contributing them to Torque simply because we needed to make sure our time was spent on tasks that brought in money in a timely fasion -- all the other indies out there know where I'm coming from on this one I'm sure.

Another example is the work I'm doing on network physics and collisions for dRacer. I've spent a good amount of time studying why when vehicles get near each other you get what JoshW calls the Torque jitter. The reason comes down to all the objects on the clients and the server having their own idea of "what time is it?". The result of all this study has been a system which I think will help propel Torque even further ahead of the competition when it comes to networking. So far I only have proof of concept working, but it's looking like this will solve a big problem with mutliplayer racing games and other games which want to have a lot of networked contact between the game objects. As BraveTree we would have a hard time monetizing this work. It would go into our games, but clearly it's value goes well beyond usage in our own games. We could have simply contributed it back to GarageGames, but as an indie you don't want to be giving away cutting edge work you do just because you don't know how to fully realize it's value. As GarageGames, the value of this project can be easily realized by simply adding the code to HEAD.

Then there's the component system, which I can now keep a more watchful eye on to make sure it conforms to my vision of what it can be. We had a deal in place before, but a lot of the work of getting it into Torque was going to fall on the GG staff, so it wasn't clear that it would end up being exactly what I think it can be. The merger allows me to take a more active role in flushing out how it will work.

Of course, the one thing the merger doesn't do is give everyone more time. So, unfortunately, not all these technology benefits are going to be realized as quickly as we'd all like. But they will happen.

Another one of Brian's concerns was that he wants tech not games, so the announcement that BT is joining GG to make more games leaves him uncertain. I strongly believe that you want us making games. That's because blue sky research is really not, in my experience, very useful. Research projects need grounding or else they get nowhere. New ideas come from new problems, and new problems come from new games. If you have someone sitting back in a chair thinking about what the issues are going to be for you guys while you make your games, you are going to get a bunch of elegant solutions to problems you don't experience, and hand wavey gestures when you bring up more mundane problems which these arm chair engineers didn't bother to think of.

Thursday Apr 21 22:45

Clark Fagot — 2005-04-21T22:45:20+00:00

Simple collider, component system, networked collision, networked physics

The last few months I've been working on a simple collision/physics system for dRacer. The basic goals of the system are that it should allow a few different simple collision image types (in the end, box and sphere), use poly soup collisions, and be network friendly. The reason for choosing box and sphere collisions only, rather than something more general like arbitrary convexes using gjk is that we only need box sphere, and we can do faster and more accurate collisions if we customize. The reason for wanting poly soup collisions, other than that they are much easier to work with (e.g. terrain convexes are, in the worst case, simply a single poly with some extra gunk added), is that they can be done fast, efficiently, and are more easily implemented than general collision systems like gjk.

You might wonder why I'd be inventing a whole new physics solution. Aren't there workable solutions out there, you might ask? Here is the landscape as I saw it:

Existing physics solutions:

1) Modified ThinkTanks collision/physics previously used in dRacer. Poly soup collisions but incomplete routines for detecting collision positions. Written with assumption that car would be on the track and we'd only be colliding with other cars and the track walls. Assumed box collisions for collisions with static objects and sphere collisions for dynamic collisions (ThinkTanks purposely used spherical body aspect to make this work).

2) Torque collision/physcis. More general convex collisions using gjk. Physics system refined and works over the net. A number of apparent issues - objects have a tendancy to get stuck, even when colliding against simple geometry like terrain (simple is in the eye of the beholder, but terrain is simple from a physics sim perspective because changes in the surface curvature are much more gradual than, say, a wall with 90 degree turns). CPU load is higher than one can tolerate for a fast moving racing game. Stacked rigid objects result in odd warping behavior. Unclear if this is due to cpu load, network bandwidth, or numerical imprecision.

3) Use ODE. ODE can handle a lot of items stacked and piled on top of each other. Problem is that it isn't networked. GG Josh has spent a fair amount of time trying to get ODE play nice over the net, and he has not had success. The big problem is that it uses a global step and this doesn't allow the network backstepping that Torque does (essentially, reseting the world to an earlier state and resuming, but only on those items which were just updated).

I wasn't really interested in tracking down all the issues to get #2 or #3 working up to the level we needed. So I started from #1. I know that sounds like more work, but what I was looking to build was a more reliable but smaller system, and I find that it is often easier to build up to this than building down from a more complex but flawed system.

Here is a video of the results (remember, this is not client only, this is network friendly):

Simple collider video - MPeg 1
Simple collider video - MPeg 2

These are the same video, just different encodings, so be easy on our server and only download one.

I also want to note that this is a component in the component system. So any object can be based off it. E.g., the race car uses the same Com3DMover component that the blocks and the sphere use. The collision image for the car is a box collider, which is the same as the blocks you see being moved around in the video.

This is nearing the end of this system as it needs to be developed for dRacer. But after dRacer I will continue development of it (and some day you should see it in some version of Torque as one of the key components in the component system). Down the line, here are some tasks I see still needing to be done:

- rest condition

Ok, so this I'll do for dRacer. Right now there is no rest condition. Because of this, the cpu load when things are stacked or are just sitting around is much higher than it needs to be. A rest condition is the cure for this ill (but notice in the video that there is no jitter even though I'm not detecting a rest condition).

- network smoothing

This I will also do for dRacer. You might notice that from time to time the update isn't entirely smooth. This is because it is networked, and the client is updated with very poor precision in order to reduce bandwidth. I was VERY surprised with how WELL the physics actually worked over the net. I expected that mis-prediction and jittery updating would be the rule, and to get around this we would have to avoid unpredictable situations (like stacks of blocks being toppled). But that wasn't the case. To get rid of the bit of jitter that is seen, I plan on adding a smooth to the render transform (say, about 250 millisecond smooth). This will be part of the mover component, so it's a feature that can be turned on and off on any object in the game. That's one of the huge benefits of the component system. Deep features like this can be shared amongst all objects very easily. Imagine implementing a similar feature on the vehicle class and now wanting to use it on the player.

- swept collisions

Currently I simply do a binary search to find collision location. This is not the fastest solution, but keeping in the spirit of the current work it is simple and fast enough for dRacer. Eventually, some swept collision checking will need to be done instead.

- more collision images

Sphere and box do the trick for now. I'd like to eventually add ellipsoid (scaled spheres), capsules (swept spheres), and tubes.

- convexes and gjk

As mentioned above I eschewed doing general convex collision because it simply isn't needed for practically any game. But for a sophisticated game engine like Torque it is nice to have the option for special cases. So eventually I'd like to add gjk to the current system. The way the components are set up now, it should be fairly straight-forward to add new collision types, including convex collisions using gjk. The main constraint is that each collision image has a priority, so higher priority images need to know about and deal with lower priority images. So one can add new collision types without re-writing the old collision types. So, in theory, one could add the current Torque gjk solution to the component system as a special case for when a more complex collision image is needed.

- unions and intersections of images

This is getting pretty far afield from where I see the system going in the near term, but there is no reason one couldn't add unions and intersections of collision images to allow one to build up to much more complex images, not unlike what is done now with Torque convex collision images.

- neighbor optimizations

One of the inefficiencies of the current system (shared by the standard Torque collision system) is that nearby objects often duplicate effort. For example, the pile of balls on the terrain in the video are all grabbing polys from the terrain. I'd like to have groups of objects enter a collision step together, sharing some of the work. This could also be useful for determining update order much like what is called a "shock step".

Sunday Feb 6 21:33

Clark Fagot — 2005-02-06T21:33:20+00:00

All the retrospectivese going on in plan files these days has got me looking back...

So much rememory (TM) going on these last few days it has gotten me looking back and thinking about how I ended up as an indie. So I thought I'd get out of character for the moment and share my experiences with others (but just this once).

What a lot of people probably don't know is that I've known Jeff for longer than pretty much anyone on this site. When I was in junior high, a couple of friends and myself began to play around with programming games in Basic and assembly language on the Apple II and C64. These two friends were Chris Cole (who later worked at Dynamix and eventually developed Chain Reaction) and Paul Bowman (who later worked for years at Dynamix and now works at Sony Bend). We used to wander into the game store that Jeff owned at the time and was staffed by, among others, Damon Slye, who later co-founded Dynamix with Jeff. The two of them looked at us through the corner of their eyes when they saw us -- they called us the "hoodlums" and thought we were trying to shoplift. :)

At the time Damon was putting the finishing touches on his first game "Stellar 7" on the Apple II (the original, not the follow-up produced years later at Dynamix which, although a decent game in it's own right, just didn't have the same charm IMO). When Jeff and Damon found out that we were aspiring programmers with some talent we all made quick friends. We would all come in and show Jeff what we had produced and he would give us comments. See, even before he had experience Jeff had an eye for what he liked and was quick to let his opinion be known (still one of his best traits). Unfortunately, the first advice Jeff ever game me was, in retrospect, bad advice. He looked at the Scramble clone I was developing and just didn't think it would be a hit. So he suggested I start over on something else. I did that, but probably should have finished what I started because it was close to my heart and I would have finished it whereas the RPG I started working on was too big and, as a result, pretty soon I started to focus more on school rather than programming. After that, I maintained contact with Jeff and Damon but never really seriously considered game development until years later.

I went to the University of Oregon and majored in Math and Computer Science. One day I took a course in Artificial Intelligence and it really hit me as interesting. The instructor suggested that if I wanted to pursue that, I should take some Psychology. I one upped her and entered a doctoral program in Psychology at the University of San Diego, California. I thoroughly enjoyed myself in graduate school, and even excelled, but 8 years later (5 years of grad school and a few years of post-docs) I started to have doubts that I really wanted to be a research psychologist all my life. I started to talk to Chris about it (he was at Dynamix at the time) and he started talking to Jeff and pretty soon they brought me back into the fold.

I started working at Dynamix in 1997 having not programmed seriously in over 10 years (I wrote Pascal programs to control psychology experiments, but they were minor projects). Even though an entry level programmer at the time, I quickly became the 3space expert, partly due to the fact that 3d math comes easy to me, but also because 3space had been written by one person and abandoned, and it fell upon me to get the library working again (and eventually re-write it). Working on Starsiege, Tribes I & II, and Trophy Hunting 4 & 5 I worked my way up the ranks at Dynamix until one day in 2001 it was closed down.

That's the day I became an Indie. I never viewed that day as a closing down of an opportunity. Rather, I figured I could do anything I wanted from that point. I could go back to Psychology, I could go into Math -- which if I were to be an academic again would make the most sense --, I could go get a job somewhere else in the industry, or I could be my own master and try to build something with the guys that I had worked with at Dynamix and trusted.

The idea of working at another game company just didn't appeal to me. At Dynamix I had always enjoyed my job, but I was also frustrated by how hard it was to be part of the decision process -- in terms of tech development, design, and simply company direction, it all came from above and I never felt master of my own destiny, just a cog in the machine.

Academia has always held a place in my heart. Both my parents were academics and I always assumed I'd do the same. But I also got a view into academic life growing up that others probably don't get. It doesn't help to have a job with total freedom and security if you aren't interested in what you are doing. Some of the people in the department my folks worked in seemed, from my perspective, to be trapped in a cushy job that was no fun for them. I had discovered that psychology didn't do it for me deep down (I enjoyed the day to day tasks, the interaction with colleagues, and find the topic incredibly interesting to this day, but it just didn't do it for me in the end). I could see going back to math, but in my late 30's I'm not sure that opportunity is still there.

So the ultimate dream job for me is really being an Indie game developer. You get all the freedom of academia and a subject matter that is eternally and inherently interesting. The only thing it lacks is security, and BraveTree is always creeping ever closer to that milestone. I hold no illusions that we don't have a long way to go, but we can clearly see the light at the end of the tunnel.

Wednesday Apr 14 17:35

Clark Fagot — 2004-04-14T17:35:41+00:00

Collaboration, racing games, and component based game objects.

About a month ago we at BraveTree central became aware of a project Brett Fattori was working on: a Wipeout style racing game. Once we saw what he had we knew we wanted to work on it with him. After some initial conversations it was clear that this was a collaboration that was going to be very fruitful.

Here is a movie of the current state of the project which Brett put together last night. You'll notice a lot of placeholder art (ThinkTanks tank with no brain on top, ThinkTanks terrain).

This project is also our first project using our new component based game object system. Anyone who has worked with the Torque vehicle and player classes has noticed that they do what they do fairly well, but if you want to change how they behave you start having to really muck with their innards. The component system is an attempt to get around this monolithic approach to game development. Instead of developing general classes with locked in feature sets, with a component based approach you build a library of components that can be combined to form all types of game classes.

For example, the car object in the movie is composed of the following components: a ComTSRender, a ComDrivingMover (derived from Com3DMover), a ComSimpleCollider (that's right, not stuck with one collision solution, you can pick and choose depending on the game), a ComCamera, and a ComLookControl (for the turret, this obviously doesn't go in the final version of the game).

The component system is built so that you can simply drop components into the BTOBjects and they know how to interact with each other. This is mostly due to BTInterfaces. E.g., the ComLook object is responsible for maintaining the look transform. It's primary task is to send horizontal and vertical angles between client and server. It also grabs a TransformInterface (which in the turret case happens to lie inside the tank shape) and sets that transform. The ComLook component is also used by the player object to control the player look animation. Since the ComLook component doesn't need to know what type of transform it is controlling, it can be reused in different contexts.

Now for some more fun. Check out this movie. That's the car on the track again but this time going 300 mph (gravity amped way up so it can stay on the track when it goes over bumps).

Next up...create the RacerMover component which will be a more refined driving model for a racing game (the current simple driving model is the ThinkTanks model with supped up parameters, but it has some limitations that make it not optimal for a racing game).

Wednesday Oct 22 21:12

Clark Fagot — 2003-10-22T21:12:58+00:00

Flirtations with the dark side -- an indie trying to get noticed.

Six months later and I still feel like we are shipping and marketing ThinkTanks (our Tank Game). We have started work on a number of other projects, but we haven't jumped into full production on any of them yet (we have always been careful that way, we do a lot of prototyping to test out ideas before jumping into full production). Instead, most of our time has been spent on making new builds, marketing, and pick up contract work (in fact, I'm being flown down to San Diego for the weekend to do some work -- hopefully I'll get to hook up with some of my old
volleyball buddies while I'm there).

ThinkTanks is now on Shockwave, RealArcade, and Reflexive. It should soon be out on a few more portals too. It has sold very well on all these portals, and we've reached customers we probably never would have reached, but we sure are giving up a big piece of the pie just to get access to new people.

We have started to think about ways to reach out to more people and bring them in to play ThinkTanks and other games at GarageGames. I wanted to share one of our ideas on how to do that.

We have created an ActiveX control that downloads and runs ThinkTanks (it also uninstalls it when they are finished so that no trace of the program is left on the players machine except what is in the internet cache). We also developed a very small version of ThinkTanks that has minimal shell, no music, and only 1 misison. This comes in at just under 3 megs. So what we have here is a web version of ThinkTanks that gives the user the experience of playing ThinkTanks in their browser (even though they aren't really doing exactly that).

So, how to leverage this? What if we could get a bunch of smaller websites (or bigger if possible) to host the web version of ThinkTanks. We could host all the files and simply supply them with a button that links directly to the game as if it's running off their site. Why would webmasters want to do this? One reason is to generate traffic to their site and make it "stickier". But why not sweeten the pot with some ad revenue like the big boys do? So we have joined the Fastclick ad network in order to serve ads to the loading page. We also offer a no ad version for those that want to just make their site stickier(and if someone thinks they can do better by finding their own sponsors we are totally open to that too).

Check out what we have put together so far: www.bravetree.com/TT/webdemo/webmasters.html. If you have any suggestions or comments please pass them on to us (either here or via webdemo@bravetree.com). We are still working on buttons and banners, so if you have a particular style or design you'd like to see let us know and we might be able to meet your needs. If you have a website, big or small, that you would want to put the control on just send us an email at webdemo@bravetree.com.

So can an indie generate an audience using gorilla tactics like this? Time will tell...

Monday Jan 6 20:16

Clark Fagot — 2003-01-06T20:16:35+00:00

Improved frame-rate for ThinkTanks 50-100% in December. Optimizations were mostly rendering speed-ups. Low end machines saw largest impact. These are worst case improvements (highest poly count scenes) -- average case seems to have improved even more.

First some numbers

Frame-rates
                                    | Before | After | After+ | After++ |
Athlon 900 | Geforce 4  | 640 x 480 |   34   |  49   |  56    |   62    |
Athlon 900 | Geforce 4  | 800 x 600 |   31   |  42   |  44    |   47    |
P1000      | Geforece 4 | 800 x 600 |   33   |  47   |  50    |   57    |
P1000      | Geforce2MX | 640 x 480 |   35   |  36   |  37    |   40    |
P1000      | Geforce2MX | 800 x 600 |   23   |  24   |  24    |   26    |
P300       | Geforce2MX | 640 x 480 |   12   |  22   |  24    |   26    |
P400       | TNT        | 800 x 600 |    8   |  12   |  12    |   14    |
P400       | TNT        | 640 x 480 |    9   |  13   |  14    |   20    |


Poly counts for 640 x 480

             | Terrain | Three-space |
Before/After |   6622  |    8250     |
After+       |   5520  |    8250     |
After++      |   2580  |    8250     |

Poly counts for 800 x 600

             | Terrain | Three-space |
Before/After |   7602  |   10774     |
After+       |   6605  |   10774     |
After++      |   3027  |   10774     |

These tables compares frame-rates (and list poly counts) for various machines before and after optimizations. The "After" condition includes several optimizations I'll mention below. The "After+" includes a terrain optimization which reduces the size of the terrain (block shift) but not so much that it affects the visual quality of our game. The "After++" reduces the terrain one more time, leaving it the precise size of our play field. This provides a nice detail setting for low end machines (and high end machines that want the best framerate without sacrificing any important visual details). ThinkTanks will use the After+ and After++ terrains only (the standard terrain is simply too large for our game).

Notice that the P400 with a TNT video card (an old card) is now in the playable range (the above numbers are worst case, standing at one end of the playfield and viewing everything -- framerate jumps considerably when in the middle of the playfield or facing away from it). This also doesn't include any detail preferences ($TS::detailAdjust, $Terrain::ScreenError and everything else is on full).

Below is a list of some of the optimizations I performed. Before you ask, yes we plan to share. Over the next couple weeks I'll post resources for some of the optimizations. Some of them will be difficult to extend to the community due to the extensiveness of the changes, but we will do our best.

1. glFinish. Go into guiCanvas.cc and get rid of the call to glFinish. It simply isn't needed. I've never been able to find any resource on the net or any book suggesting it should be used for real-time work. If you can find such a thing, let me know and I'll eat my hat. Removing this will improve framerates as much as 30% on low end machines. Note: even the "Before" framerates above had this optimization, so in a sense those numbers are understated.

2. terrCheck. SceneTraversal performs an occlusion check against the terrrain on all potentially rendered objects. I found that this was taking up about 5% of the cycles when everything was visible (we have upwards of 100 objects out there). I optimized it to use the results from the previous frame in order to decide whether to perform the check on an object that frame or wait for a few frames (16) before rechecking. Since objects tend to remain occluded or not occluded from one frame to the next this is very effective. This saved about 5% in the worst case (when nothing was really occluded) and about 4% in the case where many things were! I will post a resource on this optimization soon. Note: the terrCheck function itself is a bit odd, and probably can use a re-work. This has been submitted as a resource -- check it out.

3. cone check. The scene graph compares the bounding box of the potentially rendered object to the view frustum after already doing a bit of work (e.g., checking to see if object is fully fogged). I added a simple cone check in order to quickly accept or reject the object as being inside the view frustum. This saves the fog check and the more complicated frustum check in 90% of the cases. Savings were less than 1% on my machine, but it was clearly a positive move.

4. inlines. There are several functions that should be inlines. If you are doing a lot of ts animation you will see a benefit of inlining many of the functions in tsthread. More importantly, inline all of the FrameAllocator functions. This will get you about 2% (mostly because of the terrain, which uses the frame allocator extensively).

5. dglSetCanonicalState. Get rid of all calls to dglSetCanonicalState inside rendered objects. This will cause asserts in some cases where some object didn't reset the state properly. Just fix that problem and don't pay the price of resetting the gl state every object.

6. renderImage new. Overrode the new operator on render images. Now allocate them from a pool (be careful because interior SubObjectRenderImage has a different size than all others). This fix will give you a modest speedup, but will fight memory fragmentation which is important.

7. sleep. If you are making a single player game, consider removing the call to sleep in winWindow. The purpose of this call is to allow dedicated servers in the background to get some cycles. If you are making a single player game, that shouldn't matter (windows is a pre-emptive multi-tasking OS, so it can handle apps that don't give their cycles away to charity). This can get you a few percent (maybe more on slow machines).

8. GameTSCtrl::onRender. Remove the call to renderChildControls. The parent class already does this. Currently, all gui ctrls are being drawn twice. If any of your controls are time consuming you will get hit twice.

10. Reduce blockShift. Ok, this was a biggie for us. Added the ability to change the terrain block shift and whether or not the terrain tiles. So, by changing block shift to 7 instead of 8, we reduce the terrain block from 2048 x 2048 to 1024 x 1024. The result is fewer terrain polys and fewer textures to be blended. If you have a game like ours which is best played on a reduced playfield, this can provide a nice speedup without sacrificing any visual quality. Reducing the terrain size one more notch, brings the terrain down to 512 x 512. This is our full playfield. The low end machines sacrifice nothing important for drastically improved performance. The framerates at the top of my .plan show the drastic improvements this provides on some machines.

11. projection/viewport state. If you look at the rendering code for all the objects in the Torque you will notice all these calls that setup the projection and save off the viewport (and later restore it). Those are really a waste of time for most objects in the game (anything on the terrain rather than inside an interior). I added some state management code so that it doesn't end up doing all this work just to set up the viewport and projection matrix to be exactly like it already was. This optimmization saves about 5%.

12. state switches. One of the places where the Torque does not do a great job is the rendering pipeline. There is no management of the graphics states, so if you ever look through a gl log of a Torque rendered scene, you start seeing things like glEnable(GL_BLEND), glDisable(GL_BLEND), glEnable(GL_BLEND). You get the idea. Since most of our shapes are 3space shapes, I added a mode to the scene graph whereby it tracks whether it is in "3space mode" rather than "cannonical mode". This allows us to render all the rocks without switching state, all the trees without switching state, etc. This optimization got rid of about 8% of cpu cycles that were being spent in the state switchin code, but seemed to also create a further boost due to the remaining rendering calls taking less time. Getting this optimization to the community will be difficult. If someone wants to step forward to help out please let me know.

Ok, that's all for now. Look for some of these in resources in the coming weeks.

clark
Technical Director
BraveTree Productions