Cel shading OutLine question

#1 Look at the bottom of this page:
www.garagegames.com/docs/tgea/official/content/documentation/Materials%20and%20S...

Use the 'pass' modifier and define a second custom material.

#2 Hi Picasso

pass modifier is not working.

I did like below

new ShaderData( textureShader )
{   
	DXVertexShaderFile 	= "shaders/myshader/textureV.hlsl";   
	DXPixelShaderFile 	= "shaders/myshader/textureP.hlsl";   
	pixVersion = 2.0;
};



new ShaderData( OutLineShader )
{   
	DXVertexShaderFile 	= "shaders/myshader/outLineV.hlsl";   
	DXPixelShaderFile 	= "shaders/myshader/outLineP.hlsl";   
	pixVersion = 2.0;
};



new CustomMaterial(HeroArmorOutLine) // v1.1
{
//  mapTo = "c_armor_base_map01";
  texture[1] = "./c_armor_base_map01";

  shader = OutLineShader; 
  version = 2.0;	
 
};


new CustomMaterial(HeroArmor) // v1.1
{
  mapTo = "c_armor_base_map01";
  texture[0] = "./c_armor_base_map01";

  shader = textureShader; 
  
  pass[0] = HeroArmorOutLine;
    
 version = 2.0;	
 
};

#3 Why do you mess T0 and T1 registers?
Specify your render targets and input semantics for 2nd pass.

#4 HI Picasso
Can you tell me more detail about render targets, input semantic?

is it right?
input semantics => texture [0] ,texture[1]
render target => pass[0]


What do I need to fix below code?

new CustomMaterial(HeroArmorOutLine) 
{
 shader = OutLineShader;   
 version = 2.0;	 
};

new CustomMaterial(HeroArmor)
{  
mapTo = "c_armor_base_map01";  
texture[0] = "./c_armor_base_map01";  
shader = textureShader;     
pass[0] = HeroArmorOutLine;     
version = 2.0;	 
};

#5 Input semantics is your V registers of the GPU.
Your mapto parameter specifies the render target.
Each custom material is a single pass shader, that means it uses an input as an output from the previous pass.
That one is called a pipeline, a hardware implementation, that uses a blocks of modulation, connected withing a small pipe and a super fast cache memory, called 'fixator', That one is used to store a temporary data.

Try to watch your way, debug the shader.
Sometimes a small typo can cause a lot of problems.

#6 Picasso
I did test below code

pass[0] parameter is not working.....

BlueV.hlsl

#define IN_HLSL
#include "../shdrConsts.h"

float4x4 view_proj_matrix: register(VC_WORLD_PROJ);

struct VS_INPUT 
{
   float4 Position : POSITION0;
};

struct VS_OUTPUT 
{
   float4 Position : POSITION0;
};


VS_OUTPUT main( in VS_INPUT Input )
{
   VS_OUTPUT Output = (VS_OUTPUT) 0;


   Output.Position = mul( view_proj_matrix, Input.Position );
   
   
   return( Output );
   
}

BlueP.hlsl

struct PS_INPUT //We define the vertex out structure again
{
   float4 Position : POSITION0;
};
struct PS_OUTPUT //We define the pixel output
{
   float4 Color : COLOR0;
};
//The heart of the shader
void main(in PS_INPUT IN, out PS_OUTPUT OUT)
{
    OUT.Color = float4 (0.0f, 0.0f, 1.0f, 1.0f); //Output the color blue
}

RedV.hlsl

#define IN_HLSL
#include "../shdrConsts.h"
float4x4 view_proj_matrix: register(VC_WORLD_PROJ);

struct VS_INPUT 
{
   float4 Position : POSITION0;
};

struct VS_OUTPUT 
{
   float4 Position : POSITION0;
};

VS_OUTPUT main( in VS_INPUT Input )
{
   VS_OUTPUT Output = (VS_OUTPUT) 0;


   Output.Position = mul( view_proj_matrix, Input.Position );
   
   
   return( Output );
   
}

RedP.hlsl

struct PS_INPUT //We define the vertex out structure again
{
   float4 Position : POSITION0;
};
struct PS_OUTPUT //We define the pixel output
{
   float4 Color : COLOR0;
};
//The heart of the shader
void main(in PS_INPUT IN, out PS_OUTPUT OUT)
{
    OUT.Color = float4 (1.0f, 0.0f, 0.0f, 1.0f); //Output the color blue
}

new ShaderData( BlueShader )
{   	
DXVertexShaderFile 	= "shaders/myshader/BlueV.hlsl";   	
DXPixelShaderFile 	= "shaders/myshader/BlueP.hlsl";   	
pixVersion = 2.0;
};

new ShaderData( RedShader )
{   	
DXVertexShaderFile 	= "shaders/myshader/RedV.hlsl";   	
DXPixelShaderFile 	= "shaders/myshader/RedP.hlsl";   	
pixVersion = 2.0;
};

new CustomMaterial(BlueMaterial) 
{ 
shader = BlueShader;    
version = 2.0;	 
};

new CustomMaterial(RedMaterial)
{  
mapTo = "c_armor_base_map01";  
texture[0] = "./c_armor_base_map01";  
shader = RedShader;     
pass[0] = BlueMaterial;  //2 pass   
version = 2.0;	 
};

#7 Well, I kind of got it to work, but I don't think it does what you think it does. It drew a giant sobel filtered texture over my screen, so I had to modify the code to draw the model, but it really only sobel filtered the texture itself and not the model, there was no outline of the model's edges, but rather a model with a white texture with black lines drawn on the textures where there were edges of colors.

Is this approach wrong, or is it just my code? Can you do a model outline using shaders? Here is the shader I made from JHK's code of the first pass (the 2nd pass seems to be a clone of the first):

outlineV.hlsl

// pixVersion = 2.0;

#define IN_HLSL
#include "shdrConsts.h"

float4x4 view_proj_matrix: register(VC_WORLD_PROJ);

struct AppData
{
   float4 Pos      : POSITION;
   float2 TexCoord : TEXCOORD0;
};

struct VertexOut
{
   float4 Pos      : POSITION;
   float2 TexCoord : TEXCOORD0;
};

VertexOut main(AppData In, uniform float4x4 modelview : register(VC_WORLD_PROJ))
{
   VertexOut Out; 

   //In.Pos.xy = sign(In.Pos.xy); This didn't work either
   
   //Out.Pos = mul(In.Pos, ModelViewProjMatrix); This didn't work
   Out.Pos = mul(modelview, In.Pos);

   //Out.Pos = float4(In.Pos.xy, 0.0, 1.0); this didnt work
   Out.TexCoord = In.TexCoord; // this would do nothing I think
   
   // Image-space stuff commented out, I couldn't get this to work
   //Out.TexCoord.x = 0.5 * (1 + In.Pos.x);
   //Out.TexCoord.y = 0.5 * (1 - In.Pos.y);

   return Out;
}

outlineP.hlsl

// pixVersion = 2.0;

#define IN_HLSL
#include "shdrConsts.h"


struct VertexIn
{
   float4 Pos : POSITION;
   float2 TexCoord : TEXCOORD0;
};

struct PixelOut
{
   float4 Color    : COLOR0;
};

PixelOut main(VertexIn In, uniform sampler2D tex : TEXUNIT0) // was TEXUNIT0
{
   PixelOut Out;
   
   // One pixel offset
   const float off = 1.0 / 512.0;

   // Sample neighbor pixels
   float s00 = tex2D(tex, In.TexCoord + float2(-off, -off)).r;
   float s01 = tex2D(tex, In.TexCoord + float2( 0,   -off)).r;
   float s02 = tex2D(tex, In.TexCoord + float2( off, -off)).r;

   float s10 = tex2D(tex, In.TexCoord + float2(-off,  0)).r;
   float s12 = tex2D(tex, In.TexCoord + float2( off,  0)).r;

   float s20 = tex2D(tex, In.TexCoord + float2(-off,  off)).r;
   float s21 = tex2D(tex, In.TexCoord + float2( 0,    off)).r;
   float s22 = tex2D(tex, In.TexCoord + float2( off,  off)).r;

   // Sobel filter in X direction
   float sobelX = s00 + 2 * s10 + s20 - s02 - 2 * s12 - s22;
   // Sobel filter in Y direction
   float sobelY = s00 + 2 * s01 + s02 - s20 - 2 * s21 - s22;

   // Find edge, skip sqtexture() to improve performance ...
   float edgeSqr = (sobelX * sobelX + sobelY * sobelY);

   // ... and threshold against a squared value instead.
   Out.Color = 1.0-(edgeSqr > 0.07 * 0.07);
   return Out;
}

#8 JHK: Aha! After reading this thread probably 100 times, I finally figured out what the heck was wrong with your red/blue shader.

It's easy if you think of it like this: A CustomMaterial's texture[n] can be thought of the input texture for that shader. The mapTo parameter can be thought of as the shader's output. Therefore, each CustomMaterial that is in the multipass sequence needs to have their texture/mapTo linked together properly.

So here is a corrected version of your code (Edited texture names to be more descriptive, change back to your original names for it to work in your project):

new CustomMaterial(BlueMaterial) 
{ 
   texture[0] = "redArmor"; // this is our input, the redArmor from RedMaterial

   shader = BlueShader;    

   version = 2.0;

   mapTo = "finalArmor"; // this is the name of the texture that will be applied to the model
};

new CustomMaterial(RedMaterial)
{  
   version = 2.0;

   texture[0] = "originalArmor";  // this is your input from wherever the input armor texture was located

   shader = RedShader; 

   mapTo = "redArmor"; // we're going to output this to a temporary texture to make the pass to BlueMaterial
    
   pass[0] = BlueMaterial;  //2nd pass    
};

Hope this helps. Now let's see if we can get the outline code to work, :-)

#9 JHK: have you given up on this? :-/

I'm not sure your sobel filter will give you an edge around the model because it's only filtering the stuff drawn on the texture itself. For instance, it will draw an outline of his belt, but it won't give you a silhouette of the entire model.

Render Monkey probably used that filter as a full screen filter, so that's why it worked, but in torque shaders apply to the model. I was playing around with a resource on here to do full screen shaders, so that's the route I'm investigating now, but then the problem is that you'd have to do some special stuff:

1. First render any model that you want to have an outline in some sort of solid color. Each model must be a different color for the filter to differentiate.

2. Run the filter to get the edges of all the stuff on the screen, then save that texture.

3. Now draw all your models normally, then add the filtered edge image on top, making sure that you draw the white as transparent, and the black as opaque.

Man, the more I think about this, this is pretty intense.


The other way of doing it is to first draw the backfaces of the model in a thick black wireframe, then draw the model as normal, but I have no idea how to do that, but it somehow seems easier.