[b4bug] Outline Shader not working

#1 Eric, What's your video card?

#2 @Steve: I'm running this on a 8400M GT, on my laptop. I figure this is going to have the same effect on my workstation (8800 GT)

#3 So, anyone else with this problem? :)

#4 Don't shout I've got a hangover ... ;)
Yeah I'm seeing that not work, black sky and general mess on the objects.

FXD3D9Shader::_compileShader - Error compiling shader: E_FAIL: An undetermined error occurred (80004005)
   C:/Torque/Torque 3D 2009 Beta 4/My Projects/YorkshireRifles/game/shaders/common/particlesP.hlsl(42,24): error X3004: undeclared identifier 'prepassUncondition'
GFXD3D9Shader::_compileShader - Error compiling shader: E_FAIL: An undetermined error occurred (80004005)
   C:/Torque/Torque 3D 2009 Beta 4/My Projects/YorkshireRifles/game/shaders/common/postFx/outlineDetectP.hlsl(28,25): error X3017: 'autogenUncondition_55070f7a': cannot implicitly convert from 'const float4' to 'sampler2D'

Now go here and check your cloudlayer.

edit
Actually after saying that it looks pretty mangled in beta3 too, though that might just be my crappy 7series card.

GFXD3D9Shader::_compileShader - Error compiling shader: E_FAIL: An undetermined error occurred (80004005)
   shaders/common/postFx/outlineDetectP.hlsl(28,25): error X3017: 'autogenUncondition_55070f7a': cannot implicitly convert from 'const float4' to 'sampler2D'

#5 You need to remove the tex2D( ... ) from line 28 and leave the variables, then it'll work again.

#6 Yep Eric has it right. prepassUncondition was changed slightly in this release (or possibly in Beta3, I skipped that build). So, the new code for outlineDetectP.hlsl is:

#include "postFx.hlsl"
#include "shadergen:/autogenConditioners.h"

// GPU Gems 3, pg 443-444
float GetEdgeWeight(float2 uv0, in sampler2D prepassBuffer, in float2 targetSize)
{
   float2 offsets[9] = {
      float2( 0.0,  0.0),
      float2(-1.0, -1.0),
      float2( 0.0, -1.0),
      float2( 1.0, -1.0),
      float2( 1.0,  0.0),
      float2( 1.0,  1.0),
      float2( 0.0,  1.0),
      float2(-1.0,  1.0),
      float2(-1.0,  0.0),
   };
   
   
   float2 PixelSize = 1.0 / targetSize;
   
   float Depth[9];
   float3 Normal[9];
   
   for(int i = 0; i < 9; i++)
   {
      float2 uv = uv0 + offsets[i] * PixelSize;
      float4 gbSample = prepassUncondition( prepassBuffer, uv );
      //was prespassUncondition( tex2D(prepassBuffer, uv) )
      Depth[i] = gbSample.a;
      Normal[i] = gbSample.rgb;
   }
   
   float4 Deltas1 = float4(Depth[1], Depth[2], Depth[3], Depth[4]);
   float4 Deltas2 = float4(Depth[5], Depth[6], Depth[7], Depth[8]);
   
   Deltas1 = abs(Deltas1 - Depth[0]);
   Deltas2 = abs(Depth[0] - Deltas2);
   
   float4 maxDeltas = max(Deltas1, Deltas2);
   float4 minDeltas = max(min(Deltas1, Deltas2), 0.00001);
   
   float4 depthResults = step(minDeltas * 25.0, maxDeltas);
   
   Deltas1.x = dot(Normal[1], Normal[0]);
   Deltas1.y = dot(Normal[2], Normal[0]);
   Deltas1.z = dot(Normal[3], Normal[0]);
   Deltas1.w = dot(Normal[4], Normal[0]);
   
   Deltas2.x = dot(Normal[5], Normal[0]);
   Deltas2.y = dot(Normal[6], Normal[0]);
   Deltas2.z = dot(Normal[7], Normal[0]);
   Deltas2.w = dot(Normal[8], Normal[0]);
   
   Deltas1 = abs(Deltas1 - Deltas2);
   
   float4 normalResults = step(0.4, Deltas1);
   
   normalResults = max(normalResults, depthResults);
   
   return dot(normalResults, float4(1.0, 1.0, 1.0, 1.0)) * 0.25;
}

float4 main( PFXVertToPix IN, 
             uniform sampler2D prepassBuffer :register(S0),
             uniform float2 targetSize : register(C0) ) : COLOR0
{
   return GetEdgeWeight(IN.uv0, prepassBuffer, targetSize );//rtWidthHeightInvWidthNegHeight.zw);
}

#7 Okay ... ah, working in Beta3 - hey that's cool!

And it's working fine in NewProject Beta4 too. And it's also working fine in Warrior Camp.

here is the only things that gets mentioned in the console

==>outlinefx.enable();
C:/Torque/Torque 3D 2009 Beta 4/Genre Kits/FPS Genre Kit/game/shaders/common/postFx/outlineDetectP.hlsl(30,13): warning X3550: array reference cannot be used as an l-value; not natively addressable, forcing loop to unroll

edit: Didn't see the above post


edit2: ah yep that's what I've got now. And it's working in beta4. Notice the black clouds of Beta4.

#8 Then it's the chipset or the shader?

#9 Do you get any console errors, Eric?

Have you updated your card drivers to new ones? Done the usual stuff?

#10 By the way, outlineDetectP.hlsl and edgeDetectP.hlsl (the AA shader) are virtually identical. So if you want to setup edgeDetect.cs so that it uses the AA edge detection instead, it should atleast continue working should future engine changes modify the shader layout (so long as GG keeps the AA shader up to date as well). You just need to point PFX_OutlineEdgeDetectShader to the AA edgeDetectP shader instead of the cloned outlineDetectP shader. I only put the outline edge detection in its own file in case I needed to make changes, but didn't end up needing to.

So the new edgeDetect.cs file would look like this:

singleton GFXStateBlockData( PFX_DefaultOutlineStateBlock )
{
   zDefined = true;
   zEnable = false;
   zWriteEnable = false;
      
   samplersDefined = true;
   samplerStates[0] = SamplerClampLinear;
};

singleton ShaderData( PFX_OutlineShader )
{   
   DXVertexShaderFile 	= "shaders/common/postFx/postFxV.hlsl";
   DXPixelShaderFile 	= "shaders/common/postFx/outlineShaderP.hlsl";
         
   samplerNames[0] = "$inputTex";
   
   pixVersion = 3.0;
};

singleton ShaderData( PFX_OutlineEdgeDetectShader )
{   
   DXVertexShaderFile 	= "shaders/common/postFx/postFxV.hlsl";
   //CHANGE THE LINE BELOW
   DXPixelShaderFile 	= "shaders/common/postFx/edgeaa/edgeDetectP.hlsl";
 
   samplerNames[0] = "$inputTex";
   
   pixVersion = 3.0;
};

singleton PostEffect( OutlineFX)
{
   renderTime = "PFXAfterDiffuse";
      
   shader = PFX_OutlineEdgeDetectShader;
   stateBlock = PFX_DefaultOutlineStateBlock;
   texture[0] = "#prepass";
   target = "$outTex";
   
   new PostEffect()
   {
      shader = PFX_OutlineShader;
      stateBlock = PFX_DefaultOutlineStateBlock;
      texture[0] = "$inTex"; 
      texture[1] = "$backBuffer";
      target = "$backBuffer";
   };
};

This should help avoid any problems with future shader changes (unless of course the edge AA shader is radically changed)

#11 @Steve: my laptop is two years old and I can't update my video card because my manufacturer never released any updates. I know there's ways around it, but the last time I did such a thing, I got blue screens like it was disco.

@Matt: I'll try that right away!

#12 Sorry for the late post, been very busy. I tried your new .cs file and it works now :)