TGE Overhaul
by Mike Parker · in Torque Game Engine · 08/04/2004 (7:44 pm) · 5 replies
I sat down not very long ago with the idea that I would fiddle around with the terrain renderer. I figured going into the guts of it, ripping out that repeating terrain system and replacing it with a zone-based system would be an interesting way to spend a month or two. I know there have been other efforts in this area, but I've had Torque for 3 years now and have done next to nothing with it. I wanted to do something big!
Before too long I was looking at ways to untangle the giant ball of tightly-coupled twine that is Torque. Somewhere in there is a set of layered abstract interfaces waiting to get out. You can see it in areas like platformVideo, for example. The modules are cleanly separated structurally (by directory). But what could be a nice pyramid where the lower layers know nothing of the top layers is instead more of a collapsed pyramid where the bricks of all the layers are mixed up in a big pile.
Analogies aside, the current state of the code base makes serious modifications very difficult at best. I know people are doing it. I know TSE is, in essence, an overhaul of TGE. But there's no reason modifying Torque shouldn't be easier than it is from an architectural point of view. Trying to remove stuff you don't need usually means several hours of tracking down side-effects. Adding new stuff is pointless until you have enoough of a road map in your head to know how many places you need to add new code for a single modification.
If the modules could be cleaned up such that the boundaries are clearly defined, and that lower layers include nothing from the upper layers, then things would begin to make more sense. I'm a selfish bastard, so I'm doint this primarily for my own amusement. I have no fixed time line, no desire to retain things for which I have no need (for example, I intend to throw out all code related to any compiler except GCC), and I plan to scrap TorqueScript entirely. I'm a Python nut, so will be replacing TorqueScript with that. I dont care if it takes 1 month or 1 year, or if I never finish at all. However, if I come up with anything useful I'll happily throw out resources to the community, donate code to the GG guys for the CVS if they want it, or whatever the circumstances allow.
Before I get in too deep though I'm curious if anyone else has made any major movement in this direction. I've seen the stalled work on the terrain renderer, I'm aware of the new audio system that started as a TGE improvement and is now going into TSE. I know of Josh Ritter's PyTorque resource, which is designed to work side-by-side with TorqueScript and not to replace it. I've read about several heavy mods to the engine for various games/content packs (SpeedTree integration, shadows, lighting, RTS stuff, yadda, yadda). But I have yet to see anything about cleaning up the code base other than a few idle posts in the archives.
I've got several ideas on how I could go about it, and ultimately will settle on one of them. But the first step is mapping out the refactoring. I've been using several existing resources (documentation and diagrams) in conjunction with the latest HEAD to get a rough image. If anyone else has made any headway in this area I would be interested in hearing about it. How far did you get/have you gotten? What is/was your approach? What are/were the biggest roadblocks? That sort of thing.
So... anyone?
Before too long I was looking at ways to untangle the giant ball of tightly-coupled twine that is Torque. Somewhere in there is a set of layered abstract interfaces waiting to get out. You can see it in areas like platformVideo, for example. The modules are cleanly separated structurally (by directory). But what could be a nice pyramid where the lower layers know nothing of the top layers is instead more of a collapsed pyramid where the bricks of all the layers are mixed up in a big pile.
Analogies aside, the current state of the code base makes serious modifications very difficult at best. I know people are doing it. I know TSE is, in essence, an overhaul of TGE. But there's no reason modifying Torque shouldn't be easier than it is from an architectural point of view. Trying to remove stuff you don't need usually means several hours of tracking down side-effects. Adding new stuff is pointless until you have enoough of a road map in your head to know how many places you need to add new code for a single modification.
If the modules could be cleaned up such that the boundaries are clearly defined, and that lower layers include nothing from the upper layers, then things would begin to make more sense. I'm a selfish bastard, so I'm doint this primarily for my own amusement. I have no fixed time line, no desire to retain things for which I have no need (for example, I intend to throw out all code related to any compiler except GCC), and I plan to scrap TorqueScript entirely. I'm a Python nut, so will be replacing TorqueScript with that. I dont care if it takes 1 month or 1 year, or if I never finish at all. However, if I come up with anything useful I'll happily throw out resources to the community, donate code to the GG guys for the CVS if they want it, or whatever the circumstances allow.
Before I get in too deep though I'm curious if anyone else has made any major movement in this direction. I've seen the stalled work on the terrain renderer, I'm aware of the new audio system that started as a TGE improvement and is now going into TSE. I know of Josh Ritter's PyTorque resource, which is designed to work side-by-side with TorqueScript and not to replace it. I've read about several heavy mods to the engine for various games/content packs (SpeedTree integration, shadows, lighting, RTS stuff, yadda, yadda). But I have yet to see anything about cleaning up the code base other than a few idle posts in the archives.
I've got several ideas on how I could go about it, and ultimately will settle on one of them. But the first step is mapping out the refactoring. I've been using several existing resources (documentation and diagrams) in conjunction with the latest HEAD to get a rough image. If anyone else has made any headway in this area I would be interested in hearing about it. How far did you get/have you gotten? What is/was your approach? What are/were the biggest roadblocks? That sort of thing.
So... anyone?
#2
Actually, I am using gcc (MingW) on Windows - so the Windows stuff I am implementing first :) The platform bits will not be discarded. But I may/may not implement them. Mac and Windows are the only Indie Game markets worth supporting at present, IMO. So I doubt I will implement the refactored Linux code myself.
08/05/2004 (5:24 am)
Quote:
The only thing that I would recommend while doing this is to not through away other platform stuff (ie you are only using gcc so you are on unix or linux. keep in mind the windows stuff).
Actually, I am using gcc (MingW) on Windows - so the Windows stuff I am implementing first :) The platform bits will not be discarded. But I may/may not implement them. Mac and Windows are the only Indie Game markets worth supporting at present, IMO. So I doubt I will implement the refactored Linux code myself.
#3
Chances are, that your changes and restructuring would never ever be put into the main CVS. If so, then your code changes will very very fast become obsolete as its not possible to update automatically with head changes.
I'm just saying this from a pessimistic point of view, but I totally agree. TGE could use a lot of generic interfaces to split the code into more manageable pieces that are more generic. But its a huge project and similar to new game devs wanting to code Doom 4 as their first project imho.
But as a learning experience I bet you will become expert on TGE code structure.
08/05/2004 (5:38 am)
I would contact the GG guys first.Chances are, that your changes and restructuring would never ever be put into the main CVS. If so, then your code changes will very very fast become obsolete as its not possible to update automatically with head changes.
I'm just saying this from a pessimistic point of view, but I totally agree. TGE could use a lot of generic interfaces to split the code into more manageable pieces that are more generic. But its a huge project and similar to new game devs wanting to code Doom 4 as their first project imho.
But as a learning experience I bet you will become expert on TGE code structure.
#4
You should be sure that you understand where our priorities lie. We're focused on making sure people have the tools they need to make games. This pragmatic approach means we tend to leave well enough alone when it comes to interface and architecture.
TGE has a solid, robust, extensible architecture. People have ported it to Solaris, X-Box, and other platforms, they've ripped huge chunks out, they've made a great variety of games... Anyway, my point is that our goal is being acheived, because people are getting games made. So we're not as interested in pursuing broad sweeping architectural changes.
Architecture in an existing codebase is best changed when necessary... and otherwise left well enough alone. :)
Anyway, best of luck to you. But I thought I'd let you know that you're taking a course that's probably not going to revolutionize the TGE world. :) Would be a good learning experience, though, I bet.
08/05/2004 (9:08 am)
Funny thing, we're working with the BraveTree guys on modularizing some parts of TGE at this very moment...You should be sure that you understand where our priorities lie. We're focused on making sure people have the tools they need to make games. This pragmatic approach means we tend to leave well enough alone when it comes to interface and architecture.
TGE has a solid, robust, extensible architecture. People have ported it to Solaris, X-Box, and other platforms, they've ripped huge chunks out, they've made a great variety of games... Anyway, my point is that our goal is being acheived, because people are getting games made. So we're not as interested in pursuing broad sweeping architectural changes.
Architecture in an existing codebase is best changed when necessary... and otherwise left well enough alone. :)
Anyway, best of luck to you. But I thought I'd let you know that you're taking a course that's probably not going to revolutionize the TGE world. :) Would be a good learning experience, though, I bet.
#5
Thanks, Ben :) Honestly, this is being done for my own twisted motives. When I mentioned resources and donating code for CVS I did not mean to imply that I would say 'Here's the completely refactored Torque for CVS that should replace the existing architecture!'. What I was getting at is that if I stumble across any revelations that could prove useful in the form of resources or CVS patches, I will happily submit them (useful terrain modifications, for example). No delusions of granduer here, I promise.
Understood. You guys are doing a great job at it. My focus here is on *me* :). There's a short list of things I don't like about the engine that I am stepping back and reworking to make it more to my liking, that's all. The point of this post was not to announce my intentions for GG domination, but to find out if anyone else has attempted to solve the same problem. I had not intended to start a discussion on the merits Torque's architecture or my approach to it.
Now this is the kind of stuff I was looking for with this post. Assuming that by 'we' you mean GG, does this mean that the results of this little collaboration will eventually propagate to the HEAD? If that's the case then I can sit back and wait to see what comes out of it before I dive in on my own project. I may not even need to mess with it if you guys are doing what I hope you are.
The three major goals I am working toward are:
* Make it easy to push whole modules out into shared libraries for runtime selection. I primarily have the video and audio modules in mind here. I'm sure not looking at turning Torque into Crystal Space. But, it should be possible for someone to be able to do so without much hassle if they are so inclined.
* Eliminate couplings/dependencies that aren't needed. This goes a long way to facilitating the above. For example, the Point2I class is forward referenced in platform.h for the sole purpose of being available for the windowSize variable. Should the platform layer really be coupled to the math module this way? Wouldn't it be easier to make a WindowSize struct? What about the Core module? Is it on the same level as or is it one level above the Platform module. I see it as one level above, which means that Platform shouldn't be using Vectors. Things like that.
* Remove code which could be considered game-specific. This mostly applies to the game dirctory I think.
The way I see it, achieving these goals will allow me to maximize my use of Torque. And if the modularization you guys are working on will get me closer to it then that's even better.
EDIT: fixed a grammatical error
08/05/2004 (10:30 am)
Quote:
Anyway, best of luck to you. But I thought I'd let you know that you're taking a course that's probably not going to revolutionize the TGE world. :) Would be a good learning experience, though, I bet.
Thanks, Ben :) Honestly, this is being done for my own twisted motives. When I mentioned resources and donating code for CVS I did not mean to imply that I would say 'Here's the completely refactored Torque for CVS that should replace the existing architecture!'. What I was getting at is that if I stumble across any revelations that could prove useful in the form of resources or CVS patches, I will happily submit them (useful terrain modifications, for example). No delusions of granduer here, I promise.
Quote:
You should be sure that you understand where our priorities lie. We're focused on making sure people have the tools they need to make games. This pragmatic approach means we tend to leave well enough alone when it comes to interface and architecture.
Understood. You guys are doing a great job at it. My focus here is on *me* :). There's a short list of things I don't like about the engine that I am stepping back and reworking to make it more to my liking, that's all. The point of this post was not to announce my intentions for GG domination, but to find out if anyone else has attempted to solve the same problem. I had not intended to start a discussion on the merits Torque's architecture or my approach to it.
Quote:
Funny thing, we're working with the BraveTree guys on modularizing some parts of TGE at this very moment...
Now this is the kind of stuff I was looking for with this post. Assuming that by 'we' you mean GG, does this mean that the results of this little collaboration will eventually propagate to the HEAD? If that's the case then I can sit back and wait to see what comes out of it before I dive in on my own project. I may not even need to mess with it if you guys are doing what I hope you are.
The three major goals I am working toward are:
* Make it easy to push whole modules out into shared libraries for runtime selection. I primarily have the video and audio modules in mind here. I'm sure not looking at turning Torque into Crystal Space. But, it should be possible for someone to be able to do so without much hassle if they are so inclined.
* Eliminate couplings/dependencies that aren't needed. This goes a long way to facilitating the above. For example, the Point2I class is forward referenced in platform.h for the sole purpose of being available for the windowSize variable. Should the platform layer really be coupled to the math module this way? Wouldn't it be easier to make a WindowSize struct? What about the Core module? Is it on the same level as or is it one level above the Platform module. I see it as one level above, which means that Platform shouldn't be using Vectors. Things like that.
* Remove code which could be considered game-specific. This mostly applies to the game dirctory I think.
The way I see it, achieving these goals will allow me to maximize my use of Torque. And if the modularization you guys are working on will get me closer to it then that's even better.
EDIT: fixed a grammatical error
Torque Owner Ben Woodhead
Sounds like a big project. Keep us informed on how things are going. I would be very interested to see someone try to refactor a project this large.
The only thing that I would recommend while doing this is to not through away other platform stuff (ie you are only using gcc so you are on unix or linux. keep in mind the windows stuff).
The reason I make that recommendation is that if you do a really good job of your refactoring then it could become very useful to everybody but if its not possible to add windows and mac platforms back after your done then it can't be used. :(
Good luck.
Ben