BackBuffer Refraction
by Luigi Rosso · in Torque Game Engine Advanced · 02/26/2008 (12:11 pm) · 15 replies
Hi all,
I noticed an issue with the new water renderer. Any object that is in front of the water surface (relative to the camera) will display distortion artifacts around itself.
Here are two example screenshots, note the artifacts on the edges of the airplane:
www.realityslip.com/artifact1.jpg
www.realityslip.com/artifact2.jpg
After some investigation I found that this is due to the the refraction done by the water shader. Basically the refraction uses a copy of the framebuffer to apply texturespace distortion (based on water normals) to mimic refraction.
The problem is that the copy of the framebuffer will also contain objects above the water that could be in line of sight with the camera. If this happens, then the objects will refract as well and display the artifacts.
One solution that I was thinking about was to render another copy of the screen (much like the reflect pass does) into another render target excluding all shapebase items that are above the water (partially submerged items would still suffer the artifacting). This would definitely require rendering the terrain again although a lot of other things could be optimized out, it still seems like a heavy addition.
Another solution that I implemented in OpenGL previously was using oblique frustum clipping (http://developer.nvidia.com/object/oblique_frustum_clipping.html). This allows for an arbitrary clipping plane which would work perfect in this case as you could place it right on the water plane and even objects that are partially submerged will work. The way I'd implemented this previously was using another render target per water plane. Again costly if you have lots of water planes but not as costly as the previous solution as a lot of pixels are culled by the oblique clip which saves fill cycles.
Finally my other solution is to entirely remove refraction or just live with the issue. I'm sure others have noticed it, how have you dealt with this?
I noticed an issue with the new water renderer. Any object that is in front of the water surface (relative to the camera) will display distortion artifacts around itself.
Here are two example screenshots, note the artifacts on the edges of the airplane:
www.realityslip.com/artifact1.jpg
www.realityslip.com/artifact2.jpg
After some investigation I found that this is due to the the refraction done by the water shader. Basically the refraction uses a copy of the framebuffer to apply texturespace distortion (based on water normals) to mimic refraction.
The problem is that the copy of the framebuffer will also contain objects above the water that could be in line of sight with the camera. If this happens, then the objects will refract as well and display the artifacts.
One solution that I was thinking about was to render another copy of the screen (much like the reflect pass does) into another render target excluding all shapebase items that are above the water (partially submerged items would still suffer the artifacting). This would definitely require rendering the terrain again although a lot of other things could be optimized out, it still seems like a heavy addition.
Another solution that I implemented in OpenGL previously was using oblique frustum clipping (http://developer.nvidia.com/object/oblique_frustum_clipping.html). This allows for an arbitrary clipping plane which would work perfect in this case as you could place it right on the water plane and even objects that are partially submerged will work. The way I'd implemented this previously was using another render target per water plane. Again costly if you have lots of water planes but not as costly as the previous solution as a lot of pixels are culled by the oblique clip which saves fill cycles.
Finally my other solution is to entirely remove refraction or just live with the issue. I'm sure others have noticed it, how have you dealt with this?
About the author
#3
That's a good solution and a lot more efficient than the ones I was thinking about.
02/26/2008 (2:07 pm)
Thanks Alex,That's a good solution and a lot more efficient than the ones I was thinking about.
#4
Is this just a matter of tweaking the camera transform matrix before rendering refraction in a seperate pass, or is it more complex than that? I suppose the latter but never hurts to ask incase you're still reading ;)
02/26/2008 (3:16 pm)
Quote:
Using a separate pass for refraction with the near plane set on the water plane would be the highest quality (and slowest) solution. It's also the solution used by the Source engine.
Is this just a matter of tweaking the camera transform matrix before rendering refraction in a seperate pass, or is it more complex than that? I suppose the latter but never hurts to ask incase you're still reading ;)
#5
02/26/2008 (3:37 pm)
It's fancy tweaking of the camera transform matrix before rendering refraction in a separate pass. Eric Lengyel has some sample code and has also written many articles on the subject (e.g. Game Programming Gems 5, Section 2.6)
#6
I used that technique in another OpenGL engine. It worked out well. You can find info in this link for how to set it up:
http://developer.nvidia.com/object/oblique_frustum_clipping.html
I used it for both reflection and refractions (render everything above the plane, reflected, into a reflection buffer and render everything below the plane into a refraction buffer).
It does end up requiring rendering the scene a few times but if you have good culling it's feasible.
If I remember correctly it involved doing operations on the camera transform matrix.
Game programming gems 5 had a whole article on the algorithm.
02/26/2008 (3:40 pm)
Hey Stefan,I used that technique in another OpenGL engine. It worked out well. You can find info in this link for how to set it up:
http://developer.nvidia.com/object/oblique_frustum_clipping.html
I used it for both reflection and refractions (render everything above the plane, reflected, into a reflection buffer and render everything below the plane into a refraction buffer).
It does end up requiring rendering the scene a few times but if you have good culling it's feasible.
If I remember correctly it involved doing operations on the camera transform matrix.
Game programming gems 5 had a whole article on the algorithm.
#7
02/26/2008 (3:42 pm)
Whoops spent too much time typing :-)
#8
www.realityslip.com/clean.jpg
Now I had to make some modifications to the shaders and the framebuffer format in order to get this to work. I have a Geforce 8800 GTS so I may have made modifications that break functionality on lower end hardware. Can anyone let me know if any of these things set off alarms?
Changed framebuffer surface format from D3DFMT_X8R8G8B8 to D3DFMT_A8R8G8B8.
This one was driving me nuts, the alpha values weren't getting written to the framebuffer...turns it out it didn't have an alpha channel. Is an alpha channel in the framebuffer rarely supported or is this acceptable? This was done on ~1700 in gfxD3DDevice.cpp.
In the water pixel shader waterCubeReflectRefract.hlsl I added a conditional statement (if else). This used to be a big no no is it still? Would it be better to just lerp based on alpha value (since the method Alex described seems perfectly suited to this with since values of only 0 or 1 end up in the alpha channel).
I also had to add GFX->copyBBToSfxBuff(); call for each waterblock that gets rendered to copy the current backbuffer after having drawn the "alpha mark". I thought of a more elegant solution such as adding a new render image type for first drawing all the alpha marks then doing a copy once and then doing all the final draws. This would be beneficial when more than one waterblock is on screen, does anyone see any problems with such a solution?
02/27/2008 (9:48 am)
Works great!! Thanks for the recommendation Alex!www.realityslip.com/clean.jpg
Now I had to make some modifications to the shaders and the framebuffer format in order to get this to work. I have a Geforce 8800 GTS so I may have made modifications that break functionality on lower end hardware. Can anyone let me know if any of these things set off alarms?
Changed framebuffer surface format from D3DFMT_X8R8G8B8 to D3DFMT_A8R8G8B8.
This one was driving me nuts, the alpha values weren't getting written to the framebuffer...turns it out it didn't have an alpha channel. Is an alpha channel in the framebuffer rarely supported or is this acceptable? This was done on ~1700 in gfxD3DDevice.cpp.
In the water pixel shader waterCubeReflectRefract.hlsl I added a conditional statement (if else). This used to be a big no no is it still? Would it be better to just lerp based on alpha value (since the method Alex described seems perfectly suited to this with since values of only 0 or 1 end up in the alpha channel).
I also had to add GFX->copyBBToSfxBuff(); call for each waterblock that gets rendered to copy the current backbuffer after having drawn the "alpha mark". I thought of a more elegant solution such as adding a new render image type for first drawing all the alpha marks then doing a copy once and then doing all the final draws. This would be beneficial when more than one waterblock is on screen, does anyone see any problems with such a solution?
#9
02/27/2008 (10:13 am)
I would love to try this out Luigi. When do I modify the matrix, in addition to your changes?
#10
I actually didn't implement the Oblique Frustum Clipping method, I did the method Alex mentioned in his first post (it's different).
This method is two pass. The first pass draws the water surface writing only to the alpha channel. This is helpful as it will mark where the water is visible (depth tests will discard pixels that are out of view, behind terrain or objects like my airplane). You copy this framebuffer out to another texture to sample from when you draw the water again. So basically you do a copy to texture right after you've marked alpha values. Then you immediately draw the water again but this time using the water pixel shader. This shader needs to be modified to check if the "distort" coordinates is grabbing pixels from behind an object or from something on under the water surface. This is easy to discern as anything that is below the water will have the alpha value of 0 from the previous draw. If the value is not 1, then simply use the non-distorted coordinate. So in the end the edges of objects will not refract, like Alex said it's not perfect as you have a little halo of non-refraction around the objects but at the same time it's far less noticeable than the current system.
I'd be glad to post my changes but I was hoping someone could go through my previous post and let me know if there's anything that breaks the flexibility of the current system before giving out code. I'd also like to try to implement the second 'improvement' I listed which will only require one framebuffer copy when more than one waterblock is in view.
I'll send you an email with my changes!
02/27/2008 (10:30 am)
Hi Stefan,I actually didn't implement the Oblique Frustum Clipping method, I did the method Alex mentioned in his first post (it's different).
This method is two pass. The first pass draws the water surface writing only to the alpha channel. This is helpful as it will mark where the water is visible (depth tests will discard pixels that are out of view, behind terrain or objects like my airplane). You copy this framebuffer out to another texture to sample from when you draw the water again. So basically you do a copy to texture right after you've marked alpha values. Then you immediately draw the water again but this time using the water pixel shader. This shader needs to be modified to check if the "distort" coordinates is grabbing pixels from behind an object or from something on under the water surface. This is easy to discern as anything that is below the water will have the alpha value of 0 from the previous draw. If the value is not 1, then simply use the non-distorted coordinate. So in the end the edges of objects will not refract, like Alex said it's not perfect as you have a little halo of non-refraction around the objects but at the same time it's far less noticeable than the current system.
I'd be glad to post my changes but I was hoping someone could go through my previous post and let me know if there's anything that breaks the flexibility of the current system before giving out code. I'd also like to try to implement the second 'improvement' I listed which will only require one framebuffer copy when more than one waterblock is in view.
I'll send you an email with my changes!
#11
Perfectly acceptable and fully supported.
Lerping would be better. lerp will likely be one native instruction, if/else will be a lot more, and it doesn't help that cmp is extremely expensive, especially on older 2.0 cards.
I don't see any issues. You should be able to get away with inserting the alpha mask render into one of the existing bins; a new bin is overkill.
02/27/2008 (11:33 am)
Quote:Changed framebuffer surface format from D3DFMT_X8R8G8B8 to D3DFMT_A8R8G8B8.
This one was driving me nuts, the alpha values weren't getting written to the framebuffer...turns it out it didn't have an alpha channel. Is an alpha channel in the framebuffer rarely supported or is this acceptable? This was done on ~1700 in gfxD3DDevice.cpp.
Perfectly acceptable and fully supported.
Quote:In the water pixel shader waterCubeReflectRefract.hlsl I added a conditional statement (if else). This used to be a big no no is it still? Would it be better to just lerp based on alpha value (since the method Alex described seems perfectly suited to this with since values of only 0 or 1 end up in the alpha channel).
Lerping would be better. lerp will likely be one native instruction, if/else will be a lot more, and it doesn't help that cmp is extremely expensive, especially on older 2.0 cards.
Quote:I also had to add GFX->copyBBToSfxBuff(); call for each waterblock that gets rendered to copy the current backbuffer after having drawn the "alpha mark". I thought of a more elegant solution such as adding a new render image type for first drawing all the alpha marks then doing a copy once and then doing all the final draws. This would be beneficial when more than one waterblock is on screen, does anyone see any problems with such a solution?
I don't see any issues. You should be able to get away with inserting the alpha mask render into one of the existing bins; a new bin is overkill.
#12
I replaced the conditional with lerp.
I noticed that now the glow effects don't work. Looking into it...
// EDIT
The glow buffer uses the alpha channel and needs the write to alpha re-enabled :-) It's working now.
02/27/2008 (11:44 am)
Thanks Alex,I replaced the conditional with lerp.
I noticed that now the glow effects don't work. Looking into it...
// EDIT
The glow buffer uses the alpha channel and needs the write to alpha re-enabled :-) It's working now.
#13
From what i remember they rejected pixels from previous frame buffer by using the previous frame depth buffer. They'll be posting the slides from the GDC talks soon which covers all of their techniques for water rendering.
02/27/2008 (1:27 pm)
FYI. In Crysis they fixed this without adding a separate rendering pass just for refraction. (Note: Crysis has ~1500 draw calls per rendered frame... they can't afford more passes than they already have.)From what i remember they rejected pixels from previous frame buffer by using the previous frame depth buffer. They'll be posting the slides from the GDC talks soon which covers all of their techniques for water rendering.
#14
I ended up piggybacking the alpha marking with the Decal render bins. This actually saved an extra framebuffer copy as the refraction manager was doing this once anyway so as long as the alpha is marked before the refraction manager copies the framebuffer, you don't need to do another copy at all. So basically all this fix adds is another render pass for each waterblock. It's not as bad as it sounds as it's only writing to the alpha channel and it's not performing any heavy shader operations. I think this is pretty light considering the visual advantage.
Tom,
Interesting...I wonder if there'd be much of a gain in this case. The extra pass that this fix does is only on the water plane (literally just redrawing the water plane with no color), not redrawing anything else. It didn't seem to affect my frame rate at all. Definitely would be worth while to see their slides however. I'm sure that if the Crysis guys are using it, there's some advantage ;-)
I'd be glad to zip this up and post a resource if others want to take a look.
02/27/2008 (1:38 pm)
Alex,I ended up piggybacking the alpha marking with the Decal render bins. This actually saved an extra framebuffer copy as the refraction manager was doing this once anyway so as long as the alpha is marked before the refraction manager copies the framebuffer, you don't need to do another copy at all. So basically all this fix adds is another render pass for each waterblock. It's not as bad as it sounds as it's only writing to the alpha channel and it's not performing any heavy shader operations. I think this is pretty light considering the visual advantage.
Tom,
Interesting...I wonder if there'd be much of a gain in this case. The extra pass that this fix does is only on the water plane (literally just redrawing the water plane with no color), not redrawing anything else. It didn't seem to affect my frame rate at all. Definitely would be worth while to see their slides however. I'm sure that if the Crysis guys are using it, there's some advantage ;-)
I'd be glad to zip this up and post a resource if others want to take a look.
#15
www.garagegames.com/mg/forums/result.thread.php?qt=72596
02/29/2008 (11:45 am)
I posted the code in the private forums here:www.garagegames.com/mg/forums/result.thread.php?qt=72596
Associate Alex Scarborough
Using a separate pass for refraction with the near plane set on the water plane would be the highest quality (and slowest) solution. It's also the solution used by the Source engine.
In the Modernization Kit I used the method described in GPU Gems 2, chapter 19; which is the same method used in FarCry. It doesn't actually fix the issue though, it just inverts it. That is, in areas where a foreground object would be incorrectly refracted there is no refraction. Not perfect, but much less noticeable and it avoids rendering another pass.