Fog related translucency issues in TSE

#1 Hmm, lot of changes in MS4, this might be fixed and it might not. Wait till it comes out and repost if it's still a problem.

#2 Hey Brian,

Thank you very much for taking the time to respond... I figured that it might be fixed in MS4, I just wanted to put it out there in case it was something you guys weren't aware of.

I'll let you know if its still an issue after MS4 is released.

Keep up the good work!

Stephane

#3 I am not sure if the latest code in CVS is the 'official' MS4, but this bug is still a problem in the latest CVS code. It looks like the fog texture needs to somehow skip fogging transparent pixels... I'd be more than happy to lend a hand in fixing this... does anyone know a good place to start looking? Would it be the alpha blend mode that the fog is using? Does it need to use a different blend mode when there is transparency?

Cheers,

Stephane

#4 My guess is around renderObject() in TSStatic.

#5 Hmm, well you'd ideally want to change the procedural shader generation. The fog uses a lerp function between no fog and fog. You need it to take into account the alpha value of the pixel being shaded in question.

It would be easier to just create your own CustomMaterial and shader for this piece of geometry.

#6 @Stefan: I don't think this is a TSStatic specific bug. I haven't tried, but there is nothing that would lead me to believe that this isn't also an issue with StaticShapes, ShapeBases, Players, Interiors, etc.

I think this issue is something more generic. Thanks for the tip though...

@Brian: I think you've spotted the source of the issue. As you get closer to the shape, the fog obviously dissipates and the transparent pixels are no longer fogged. This raises an interesting point though... How should one handle fogging transparent pixels in a general sense? How does TGE handle this? The project I am working on is all content ported from TGE, where everything works just great... so doing things the same way as they are done in TGE would be ideal.

But how does TGE handle fogging transparent pixels? I guess the easiest solution for now would be to modify the shader to not fog any pixels that are 100% transparent... and maybe make that percentage variable so that it could be fine-tuned on a per material basis or something...

I'd love some feedback though from someone who might know more about this than me...

Thank you very much Brian and Stefan for your responses!

Stephane

#7 Okay, I gave this some more thought... and here's the conclusion I came to...

You shouldn't be changing the transparency value of the original pixel. I believe the TSE fogging method changes the transparency value of the fogged pixel to match the fog amount. This shouldn't happen. The fog amount should be used only to set the color of the fog at that pixel and not the transparency of the pixel.

Someone please correct me if I am going down the wrong path...

Thanks,

Stephane

#8 Hey!! What do you know! I might have actually been right! :)

Here are my changes:

void FogFeat::processPix(Vector<ShaderComponent*>& componentList, GFXShaderFeatureData& fd)
{
   // create texture var
   Var* fogMap = new Var;
   fogMap->setType("sampler2D");
   fogMap->setName("fogMap");
   fogMap->uniform = true;
   fogMap->sampler = true;
   fogMap->constNum = Var::getTexUnitNum();     // used as texture unit num here
   
   // grab connector texcoord register
   ConnectorStruct* connectComp = dynamic_cast<ConnectorStruct*>(componentList[C_CONNECTOR]);
   Var *fogCoord = connectComp->getElement(RT_TEXCOORD);
   fogCoord->setName("fogCoord");
   fogCoord->setStructName("IN");
   fogCoord->setType("float2");
   fogCoord->mapsToSampler = true;
   
   Var* fogColor = new Var;
   fogColor->setType("float4");
   fogColor->setName("fogColor");
   LangElement* fogColorDecl = new DecOp(fogColor);
   
   // Create a floating point variable to temporarily store the pixel's alpha value
   [b]Var* colorA = new Var;
   colorA->setType("float");
   colorA->setName("colorA");
   LangElement* colorADecl = new DecOp(colorA);[/b]


   MultiLine* meta = new MultiLine;
   
   meta->addStatement(new GenOp("   @ = tex2D(@, @);\r\n", fogColorDecl, fogMap, fogCoord));

   // if no color, then fog is only feature in pass
   Var* color = (Var*) LangElement::find("col");
   if (!color)
      meta->addStatement( new GenOp( "   @;\r\n", assignColor( fogColor ) ) );
   else
   {
      if (fd.features[GFXShaderFeatureData::DynamicLight])
         meta->addStatement(new GenOp("   @.a *= 1.0 - @.a;\r\n", color, fogColor));
      else
      {
         [b]// Store the old alpha value of the pixel
         meta->addStatement(new GenOp("   @ = @.a;\r\n", colorADecl, color));
         // Perform the fogging calculation (a simple interpolation)[/b]
         LangElement* statement = new GenOp("lerp( @, @, @.a )", color, fogColor, fogColor);
         [b]// Store the newly fog-tinted pixel[/b]
         meta->addStatement(new GenOp("   @ = @;\r\n", color, statement));
         [b]// Restore the old alpha value from the pixel
         meta->addStatement(new GenOp("   @.a = @;\r\n", color, colorA));[/b]
      }
   }

   output = meta;
   return;
}

There is probably a cleaner method of doing this... rather than creating a variable to store the alpha value, one could just omit the alpha value from the calculation in the first place. I am new to shader coding and just chose the easiest way...

Let me know if anyone knows if this would cause any issues down the road...

Cheers,

Stephane

Edit: Edited spacing

#9 I think all you need to do is change the lerp statement to only lerp the .rgb values. Ie, the output should look something like:

OUT.col.rgb = lerp( OUT.col.rgb, fogColor.rgb, fogColor.a );


If you could test that for me Stephane, then I'll check in the fix.

#10 Hey Brian,

Yep, your suggestion works just as well. I just wasn't sure what the syntax was to get just the rgb values.

Replace the following in shaderFeature.cpp (approx. lines 1073 & 1074):

LangElement* statement = new GenOp("lerp(@, @, @.a)", color, fogColor, fogColor);
  meta->addStatement(new GenOp("   @ = @;\r\n", color, statement));

With this:

LangElement* statement = new GenOp("lerp(@.rgb, @.rgb, @.a)", color, fogColor, fogColor);
  meta->addStatement(new GenOp("   @.rgb = @;\r\n", color, statement));

#11 Cool, thank you.