Detailed Questions on new Shader Engine???
by Laurence Grant · in Torque Game Engine · 03/14/2006 (7:00 am) · 13 replies
Hi everyone,
Ive been looking at Ogre, Irrlicht and now Torque. Ive been developing for over 25 years, and Im a very experienced 3D/DirectX developer, although Ive never worked professionally in the industry, its been my hobby for a long time. Ive been working on my own engine for a long time, but my real passion is the game Ive envisioned for many years. Ive decided the current technology has reached a satisfactory level and in wanting to really focus on my game, Ive decided to spend time looking at the latest (affordable) engines on the market to expedite my development efforts. (I know you all wanted to hear my life story ;-) )
Questions:
1. Im Interested in the new Shader Engine. I assume this is still a portal based engine?
2. I need to find out more details on how the portals are implemented?
a. I want to have each building as a separate collection of portals. Instead of a fixed LEVEL
b. I want to be able to add and remove buildings in-game on the fly
c. I would even like a paging type algorithm like terrain, where huge buildings and complexes can be loaded in stages instead of all at once.
d. Ideally it would be modular, allowing me to add models together (on demand) to form larger complexes
e. If Im outdoors and rendering a building, and the building has portal based rooms in it, I want to manage the depth of the portal loop (for performance) to not iterate as deep on far away buildings. Is this possible?
f. Ideally, portals would be collections of rooms that are each separate objects and a linkage that defines how they connect, so the portal is a list of room model ids and the linkage of each opening in each room. Then I could change the linkage on demand, and only need to load the linkage into memory. If the linkage ties many different objects, I dont need to actually load the object until the renderer decides to render it. Any plans on this (even better if its in there already?
3. I want to find out more details on the new paging terrain?
a. Can I move between terrain, space and back to another planet relatively smoothly, without incurring load delays? In other words, since the terrain is paging, I expect the performance to be very quick. Therefore, I assume I can change height maps (a new planet) on demand?
b. Can terrain be deformed in-game and saved out? I plan on storing my terrain on the server, and only downloading the parts of the planet the player is on. If a large explosion damages the terrain I want to store that in the DB so all players see it the next time they visit that part of the planet.
c. Is the paging implemented through a splitting technique that is done at development time? For large planets I expect to have 10s of thousands of height maps the comprise the planet but only need 50-100 at run-time for rendering, so I only want to maintain a few hundred (caching) locally and pull down what I need from the server when its needed.
d. Can you build caves into the side of the terrain? Is there a way to make a hole?
e. Can you have outcrops or non height map shapes in the terrain? Does this have to be implemented with models on top of the terrain?
f. Could I truly map a life sized planet to scale (not worried about disk space)
I know thats a lot of questions, and I apologize, but Im really close to pulling the trigger and licensing both the TGE and TSE and if I could get a comfort level on the above questions Ill finalize the order today.
--- post was too large. See part 2 for a few additional questions
If this is the wrong forum for these please let me know.
Thanks
Ive been looking at Ogre, Irrlicht and now Torque. Ive been developing for over 25 years, and Im a very experienced 3D/DirectX developer, although Ive never worked professionally in the industry, its been my hobby for a long time. Ive been working on my own engine for a long time, but my real passion is the game Ive envisioned for many years. Ive decided the current technology has reached a satisfactory level and in wanting to really focus on my game, Ive decided to spend time looking at the latest (affordable) engines on the market to expedite my development efforts. (I know you all wanted to hear my life story ;-) )
Questions:
1. Im Interested in the new Shader Engine. I assume this is still a portal based engine?
2. I need to find out more details on how the portals are implemented?
a. I want to have each building as a separate collection of portals. Instead of a fixed LEVEL
b. I want to be able to add and remove buildings in-game on the fly
c. I would even like a paging type algorithm like terrain, where huge buildings and complexes can be loaded in stages instead of all at once.
d. Ideally it would be modular, allowing me to add models together (on demand) to form larger complexes
e. If Im outdoors and rendering a building, and the building has portal based rooms in it, I want to manage the depth of the portal loop (for performance) to not iterate as deep on far away buildings. Is this possible?
f. Ideally, portals would be collections of rooms that are each separate objects and a linkage that defines how they connect, so the portal is a list of room model ids and the linkage of each opening in each room. Then I could change the linkage on demand, and only need to load the linkage into memory. If the linkage ties many different objects, I dont need to actually load the object until the renderer decides to render it. Any plans on this (even better if its in there already?
3. I want to find out more details on the new paging terrain?
a. Can I move between terrain, space and back to another planet relatively smoothly, without incurring load delays? In other words, since the terrain is paging, I expect the performance to be very quick. Therefore, I assume I can change height maps (a new planet) on demand?
b. Can terrain be deformed in-game and saved out? I plan on storing my terrain on the server, and only downloading the parts of the planet the player is on. If a large explosion damages the terrain I want to store that in the DB so all players see it the next time they visit that part of the planet.
c. Is the paging implemented through a splitting technique that is done at development time? For large planets I expect to have 10s of thousands of height maps the comprise the planet but only need 50-100 at run-time for rendering, so I only want to maintain a few hundred (caching) locally and pull down what I need from the server when its needed.
d. Can you build caves into the side of the terrain? Is there a way to make a hole?
e. Can you have outcrops or non height map shapes in the terrain? Does this have to be implemented with models on top of the terrain?
f. Could I truly map a life sized planet to scale (not worried about disk space)
I know thats a lot of questions, and I apologize, but Im really close to pulling the trigger and licensing both the TGE and TSE and if I could get a comfort level on the above questions Ill finalize the order today.
--- post was too large. See part 2 for a few additional questions
If this is the wrong forum for these please let me know.
Thanks
#2
Wow. Lots of questions. Unfortunately, most of them are way beyond me. But I'll try to help out with a few of them. And in the process, maybe I can help bump thread this up a little higher. ;)
3. a. I don't think TSE can handle space to surface transitions natively. But after reading your post, I started thinking that a skybox/skysphere paging system might provide a decent solution (with a fair bit of work). That's just a guess though.
3. b. Currently TSE terrain can't be deformed in-game. That will probably change soon.
3. d. I don't think TSE currently has a hole function for Atlas terrains. Again, that will probably be added in at a future milestone.
3. e. If I understand your question correctly, yes. You can place almost any kind of mesh/polysoup object (as a .dts) onto or into your terrains using the built-in world editor or through script/triggers.
3. f. According to Ben Garney, TSE's Atlas terrain system can represent more land than is visible in the "observable universe". I think I got that quote right. :) The only limits seem to be your available storage space.
4. b. From what I've read in other posts, it is possible to rip out Torque's networking and replace it with something else. Generally, when this comes up, someone asks why anybody would want to do something that drastic, but I see where you're coming from.
Ok. That's all I can offer. I'm very new to TSE, so my knowledge of it is very limited. But hopefully this will help some.
Aaron E.
[Edit: typo]
03/14/2006 (8:06 am)
Laurence,Wow. Lots of questions. Unfortunately, most of them are way beyond me. But I'll try to help out with a few of them. And in the process, maybe I can help bump thread this up a little higher. ;)
3. a. I don't think TSE can handle space to surface transitions natively. But after reading your post, I started thinking that a skybox/skysphere paging system might provide a decent solution (with a fair bit of work). That's just a guess though.
3. b. Currently TSE terrain can't be deformed in-game. That will probably change soon.
3. d. I don't think TSE currently has a hole function for Atlas terrains. Again, that will probably be added in at a future milestone.
3. e. If I understand your question correctly, yes. You can place almost any kind of mesh/polysoup object (as a .dts) onto or into your terrains using the built-in world editor or through script/triggers.
3. f. According to Ben Garney, TSE's Atlas terrain system can represent more land than is visible in the "observable universe". I think I got that quote right. :) The only limits seem to be your available storage space.
4. b. From what I've read in other posts, it is possible to rip out Torque's networking and replace it with something else. Generally, when this comes up, someone asks why anybody would want to do something that drastic, but I see where you're coming from.
Ok. That's all I can offer. I'm very new to TSE, so my knowledge of it is very limited. But hopefully this will help some.
Aaron E.
[Edit: typo]
#3
i dont think any current systems can support something like this. The best you could probably achieve is several large world "zones" which together would comprise something like what you would want. actually I dont think it would be the terrain or heightmaps so much as remembering the positions of so many terrestial objects, unless you plan on having a planet of barren terrain.
03/14/2006 (8:25 am)
"Could I truly map a life sized planet to scale (not worried about disk space)"i dont think any current systems can support something like this. The best you could probably achieve is several large world "zones" which together would comprise something like what you would want. actually I dont think it would be the terrain or heightmaps so much as remembering the positions of so many terrestial objects, unless you plan on having a planet of barren terrain.
#4
Pagings buildings/objects/paths/etc is not currently implemented. I played with implementing something similar about a year ago in TGE. I had node-based interiors loading and unloading as needed. The segments were small and did not impact the engine load times. However, I did run into problems with patthing across instances of interiors and such. It reminded me of running from the Nemesis in Resident Evil 3 and hitting the safe rooms. If you made it there, Nemesis would magically disappear to show up elsewhere later. Except my paths just died. I could have probably have worked on it a little more, but the un/loading scheme was a horrible hack in the first place as I was wondering if I could do it rather than trying to do it "right". I realized while working on it that it had little to no functional relevance to my prototyping projects at the time.
Sorry I can't be more helpful. Most of my work is still in TGE rather than TSE.
03/14/2006 (8:27 am)
You get the complete source, so the "can I" questions are dependent on your skills and how fast it takes you to get up to speed with a complex engine (assuming no help, which of course, is an insane assumption...though a worst-case scenario one). It sounds like you already have a strong base of experience to start from, so I'd expect you to pick it up and run much faster than many hopefuls.Pagings buildings/objects/paths/etc is not currently implemented. I played with implementing something similar about a year ago in TGE. I had node-based interiors loading and unloading as needed. The segments were small and did not impact the engine load times. However, I did run into problems with patthing across instances of interiors and such. It reminded me of running from the Nemesis in Resident Evil 3 and hitting the safe rooms. If you made it there, Nemesis would magically disappear to show up elsewhere later. Except my paths just died. I could have probably have worked on it a little more, but the un/loading scheme was a horrible hack in the first place as I was wondering if I could do it rather than trying to do it "right". I realized while working on it that it had little to no functional relevance to my prototyping projects at the time.
Sorry I can't be more helpful. Most of my work is still in TGE rather than TSE.
#5
The Terrain system I built is capable of unlimited terrain sizes, limited only by disk. Since it's using a paging technology too, it loads in sub second time and has no perceivable degradation as you travel the planet. My current largest planet is the size of the US, but that's only because that's how large my height map was. However, it was done as a prototype originally and needs work. If TSE has something comparable, I'd just as well use it, and it sounds like it is.
As for foliage and not having a barren terrain, that is where I need the ability to dynamically add and remove content on the fly, and it sounds like I can do that, right? My engine uses a hierarchal structure to define and store the objects for the entire planet. Then it performs distance checks and only traverses the tree for those zones that are close enough. Also depending on distance it can use a billboard to show an entire forest, change to individual billboards when you get closer then change to high detailed models when you get closer. I'm assuming I can implement something similar in TSE?
TSE and TGE both must have a scene manager of some sort. To perform what I described above, I would be relying on the scene manager for the rendering, but I'd possibly be removing and adding items into the scene manager every frame (or every "number" of frames, based on optimization options). This is doable, right?
David, you are right, with the source I can do anything, but my intent is to work on the game not the engine, so I don't want to have to become a guru with the Torque internals any more than I have too. There is enough work to do on the game, and for now I'm just a one person team. A friend helps when he can, but he's got his own things going on.
03/14/2006 (8:48 am)
Thanks to all, all great feedback. Please keep it coming. The more details and opinions the better!The Terrain system I built is capable of unlimited terrain sizes, limited only by disk. Since it's using a paging technology too, it loads in sub second time and has no perceivable degradation as you travel the planet. My current largest planet is the size of the US, but that's only because that's how large my height map was. However, it was done as a prototype originally and needs work. If TSE has something comparable, I'd just as well use it, and it sounds like it is.
As for foliage and not having a barren terrain, that is where I need the ability to dynamically add and remove content on the fly, and it sounds like I can do that, right? My engine uses a hierarchal structure to define and store the objects for the entire planet. Then it performs distance checks and only traverses the tree for those zones that are close enough. Also depending on distance it can use a billboard to show an entire forest, change to individual billboards when you get closer then change to high detailed models when you get closer. I'm assuming I can implement something similar in TSE?
TSE and TGE both must have a scene manager of some sort. To perform what I described above, I would be relying on the scene manager for the rendering, but I'd possibly be removing and adding items into the scene manager every frame (or every "number" of frames, based on optimization options). This is doable, right?
David, you are right, with the source I can do anything, but my intent is to work on the game not the engine, so I don't want to have to become a guru with the Torque internals any more than I have too. There is enough work to do on the game, and for now I'm just a one person team. A friend helps when he can, but he's got his own things going on.
#6
I have a feeling we will know a little more about the advancements GG has made in the milestone list next week at GDC.
03/14/2006 (9:45 am)
It sounds like your engine is rather robust already. But I can easily see the benefit of licensing another engine which hopefully has what you want to get away from being an engine developer. Unfortunately there are a number of implementation specific needs in your engine which you would have to add to TSE such as the dynamic paging of objects (interiors, models, paths, triggers, effects, etc). Currently ATLAS supports a rather huge file (limited by the size of JPEG specification, though that limitation will go away in future milestones with the addition of multiple images and other formats.I have a feeling we will know a little more about the advancements GG has made in the milestone list next week at GDC.
#7
torque wasn't designed to handle dynamic creation/removal of structures. structures can be transformed and manipulated at runtime, but theyre supposed to all be pre-loaded and pre-lit. if this is a requirement for your game, you will probably need to make engine changes, but you could always try to find some kind of workaround in script. the good news is torque uses a network method for culling out distant objects so you don't need to worry about the engine doing checks for structures halfway across the world...afaik.
03/14/2006 (9:51 am)
"As for foliage and not having a barren terrain, that is where I need the ability to dynamically add and remove content on the fly, and it sounds like I can do that, right? "torque wasn't designed to handle dynamic creation/removal of structures. structures can be transformed and manipulated at runtime, but theyre supposed to all be pre-loaded and pre-lit. if this is a requirement for your game, you will probably need to make engine changes, but you could always try to find some kind of workaround in script. the good news is torque uses a network method for culling out distant objects so you don't need to worry about the engine doing checks for structures halfway across the world...afaik.
#8
There is a plugin for RTS games. RTS games allow you to create facilities on the fly. In an RTS you aren't entering the facility, but you are dynamically adding and removing content. It's the same idea, only I want to extend it to a structure I could actually enter. Is it really that different?
03/14/2006 (10:05 am)
Sean,There is a plugin for RTS games. RTS games allow you to create facilities on the fly. In an RTS you aren't entering the facility, but you are dynamically adding and removing content. It's the same idea, only I want to extend it to a structure I could actually enter. Is it really that different?
#9
:-)
03/14/2006 (10:07 am)
PS: There is a lot that Torque has to offer, hense my interest in trying to find a way to make it work, but I don't want invest $250 and a few months of work to find out it just not as flexible as I would have hoped.:-)
#10
03/14/2006 (10:29 am)
I believe all of the units available for a given map are preloaded with the RTS kit--I haven't actually looked that deeply...and now I feel really stupid for not looking when I was playing with this a long while back. The "adding" or "swapping" allows for an extension of the world editor's add/remove functionality for DTS objects but includes a number of more-friendly features which RTS users are used to. But I believe everything is preloaded.
#11
03/14/2006 (10:36 am)
I hope no one minds, I'm going to post a new thread on preloading vs dynamic loading. I don't the question to be lost in this thead because people can't can't past to first part. 
#12
2. Portals are manually placed by artists when they build Interiors (buildings and structures). The basic idea is that they seal the room(s) with portal brushes covering the openings. If a portal brush is in the player's view then the engine renders the geometry of the "zones" that are on either side of the portal and whatever objects are inside that zone. For example, the big huts in the demo mission have a portal across their doors. If you are behind the buildings then the portal brushes aren't in view and the geometry inside the huts (like the fireplace) aren't getting rendered. Also, the crossbows and ammo inside the huts aren't getting rendered. As you come around the hut the portal brush will come into view and now everything inside that interior zone will start to get rendered. You *can* have multiple zones inside an Interior defined by several sets of portal brushes.
a. This is how it works.
b. As far as portals goes this is fine. But you are going to run into a problem in that the removed buildings will leave shadows on the terrain and surrounding buildings and the added buildings will not be lit. Lightmapping is expensive and it is going to take some work to get it to happen smoothly in the background. Currently, in TGE 1.4 we have it so that newly added Interiors will at least be shaded by the sun. TSE will most likely have options for dynamic lighting on buildings (I have stencil shadows ready to go) but that can get costly in terms of rendering performance.
c. See my reply to your other thread. Short summary: somewhat possible/already done.
d. The engine has no major issues with you combining other objects together (there is a bug with overlapping zoned/portalled Interiors but that is going to get fixed...the lighting can get a little odd at times).
e. The engine already mostly handles this correctly as I outlined above. You are going to want to make sure that your buildings are modelled with portals/zones in mind (like having foyers or halls that turn just inside an exterior door).
f. Portals define geometry sealed zones. Any object inside the zone will automatically not be rendered until the portals attached to that zone are in view of the camera. I believe the zones can also effect the networked scoping which controls when objects get loaded.
3. Reading Ben's .plans is a good place to start.
a. Yes. If you have the hard drive space to handle this then Atlas can.
b. In the works.
c. The Atlas terrains are broken up into LODed chunks at development time. Having these stream from the server is something you would have to implement yourself.
d. Coming soon.
e. This will be possible in the future.
f. Yes. There are a few bugs in the current version that would make this difficult (but not impossible) but Ben has worked through them in the new version.
4.
a. It is very difficult to remove Torque's networking from the engine since the engine was built around it from the ground up and nearly every part of it is involved or reliant on it. You can *work with* Torque's networking and put a nice database or server backend behind it to handle the meta-networking tasks (data storage, matching making, server/zone management, load balancing). Several people have been very successful doing this (take a look at Minions of Mirth. Gutting Torque's networking is a far, far hard and ultimately useless task.
b. You have the source code so you technically don't have to but as I detailed above it is a *really* good idea to work with the existing networking structure.
c. See above.
5. Torque supports multi-threading. What you do with it is your call =)
a. Sure, why not.
03/14/2006 (12:09 pm)
1. Yes2. Portals are manually placed by artists when they build Interiors (buildings and structures). The basic idea is that they seal the room(s) with portal brushes covering the openings. If a portal brush is in the player's view then the engine renders the geometry of the "zones" that are on either side of the portal and whatever objects are inside that zone. For example, the big huts in the demo mission have a portal across their doors. If you are behind the buildings then the portal brushes aren't in view and the geometry inside the huts (like the fireplace) aren't getting rendered. Also, the crossbows and ammo inside the huts aren't getting rendered. As you come around the hut the portal brush will come into view and now everything inside that interior zone will start to get rendered. You *can* have multiple zones inside an Interior defined by several sets of portal brushes.
a. This is how it works.
b. As far as portals goes this is fine. But you are going to run into a problem in that the removed buildings will leave shadows on the terrain and surrounding buildings and the added buildings will not be lit. Lightmapping is expensive and it is going to take some work to get it to happen smoothly in the background. Currently, in TGE 1.4 we have it so that newly added Interiors will at least be shaded by the sun. TSE will most likely have options for dynamic lighting on buildings (I have stencil shadows ready to go) but that can get costly in terms of rendering performance.
c. See my reply to your other thread. Short summary: somewhat possible/already done.
d. The engine has no major issues with you combining other objects together (there is a bug with overlapping zoned/portalled Interiors but that is going to get fixed...the lighting can get a little odd at times).
e. The engine already mostly handles this correctly as I outlined above. You are going to want to make sure that your buildings are modelled with portals/zones in mind (like having foyers or halls that turn just inside an exterior door).
f. Portals define geometry sealed zones. Any object inside the zone will automatically not be rendered until the portals attached to that zone are in view of the camera. I believe the zones can also effect the networked scoping which controls when objects get loaded.
3. Reading Ben's .plans is a good place to start.
a. Yes. If you have the hard drive space to handle this then Atlas can.
b. In the works.
c. The Atlas terrains are broken up into LODed chunks at development time. Having these stream from the server is something you would have to implement yourself.
d. Coming soon.
e. This will be possible in the future.
f. Yes. There are a few bugs in the current version that would make this difficult (but not impossible) but Ben has worked through them in the new version.
4.
a. It is very difficult to remove Torque's networking from the engine since the engine was built around it from the ground up and nearly every part of it is involved or reliant on it. You can *work with* Torque's networking and put a nice database or server backend behind it to handle the meta-networking tasks (data storage, matching making, server/zone management, load balancing). Several people have been very successful doing this (take a look at Minions of Mirth. Gutting Torque's networking is a far, far hard and ultimately useless task.
b. You have the source code so you technically don't have to but as I detailed above it is a *really* good idea to work with the existing networking structure.
c. See above.
5. Torque supports multi-threading. What you do with it is your call =)
a. Sure, why not.
#13
03/14/2006 (12:47 pm)
Thanks again Matt. This was great. Just what I was looking for. 
Torque Owner Laurence Grant
---------
4. Networking for MMOPG
a. I know your networking is intended for multi-play, not MMO. Ive already implemented my own networking system with a C++ multi-process/multi-threaded server and a third-tier database backend. Is there anything preventing, limiting or complicating my ability to use my own networking?
b. Am I required to integrate into your existing networking infrastructure
c. Would it require a major disruption to the existing architecture?
5. I assume I can write mutli-threaded clients? Knowing I should keep all rendering local to one thread, but I want another thread for outgoing packets, for incoming packets and a 4th thread for all keyboard and control input.
a. Since your engine provides the input controls, can I break it out to my own thread?