Bad fog on scaled objects

#1 Problem since MS4, see this post: www.garagegames.com/mg/forums/result.thread.php?qt=57923

Wish they fixed it.

#2 I'll try to peek at it soon, the issue should be in FogFeat::processVert, it probably needs to do something like this:

In HLSL:
float4x4 unscaledTrans = objTrans;
trans[3][0] = 1;
trans[3][1] = 1;
trans[3][2] = 1;
float3 transPos = mul( unscaledTrans, IN.position );

In ShaderGen:

Replace:
meta->addStatement( new GenOp( "\r\n // fog setup\r\n @ = mul( @, @ );\r\n", transPosDecl, objTrans, position ) );

With:
meta->addStatement( new GenOp( "\r\n // fog setup\r\n float4x4 unscaledTrans = objTrans;\r\n trans[3][0] = 1; \r\n trans[3][1] = 1; \r\n trans[3][2] = 1; \r\n float3 @ = mul( @, @ );", transPosDecl, objTrans, position ) );

I haven't tested this, but it'll be something like this. I'll get a test setup soon and confirm.

#3 Thanks for the support! :)

#4 I still haven't gotten over to this, but I'm working on related things, and the bug could be here:

In tsMesh.cpp TSMesh::render, look for this:

// setup transforms
MatrixF objTrans = GFX->getWorldMatrix();
MatrixF proj = GFX->getProjectionMatrix();
MatrixF world;

// Kill scale here!

world.mul( smCamTrans, objTrans );

if (getFlags(Billboard))
forceFaceCamera(world);

proj.mul( world);
proj.transpose();

#5 I've been messing around with this...

First, you need to move the "Kill scale here" location to below the "world.mul( smCamTrans, objTrans );" line. You want scale in that world matrix, or your object completely loses it's scale. But after that calculation, removing scale from objTrans is probably fine as long as it's not used for anything else that needs scale.

But you can't kill the scale by just setting some matrix values to 1, unless the object has no rotation. Fortunately, I'm pretty sure that at this point, objTrans is simply a scaled version of the original object's render-transform. Therefore we can make objTrans an unscaled matrix just by resetting it to the render-transform.

I've been testing this code and it generally appears to work correctly, (from tsMesh.cpp TSMesh::render like above):

// setup transforms
   MatrixF objTrans = GFX->getWorldMatrix();
   MatrixF proj     = GFX->getProjectionMatrix();
   MatrixF world;

   world.mul( smCamTrans, objTrans );

   [b]// Kill scale here!
   if (smObject)
     objTrans = smObject->getRenderTransform();[/b]

   if( getFlags(Billboard) )
   {
      if (getFlags(BillboardZAxis))
      {
         forceFaceCameraZAxis(world);  
      }
      else
      {
         forceFaceCamera(world);  
      }
   }

   GFX->pushWorldMatrix();
   GFX->setWorldMatrix( world );
   proj.mul( world);
   proj.transpose();

Anyone see any problem with this?

One thing it does not appear to fix is scaling due to animated dts scale, but I suspect that will require some special treatment.

#6 You, my good sir, own.

#7 Great find Jeff

#8 Hey Jeff,

We found some lighting artifacts with the fix above. Here's the fix we came up with here (modded for TGEA):
We're still testing it, but it's looking good.

Find the sections of code that look like this:

// fill in eye data
   Point3F eyePos = gRenderInstManager.getCamPos();
   objTrans.mulP( eyePos );
   GFX->setVertexShaderConstF( VC_EYE_POS, (float*)&eyePos, 1 );

Replace with this:

// fill in eye data
   Point3F eyePos = gRenderInstManager.getCamPos();
   objTrans.mulP( eyePos );
   eyePos.convolveInverse(objTrans.getScale());
   GFX->setVertexShaderConstF( VC_EYE_POS, (float*)&eyePos, 1 );

#9 'getScale' : is not a member of 'MatrixF'

According to my error output.

#10 Here's an implementation of MatrixF::getScale you can add:

inline Point3F MatrixF::getScale() const
{
Point3F scale;
scale.x = mSqrt(m[0]*m[0] + m[4] * m[4] + m[8] * m[8]);
scale.y = mSqrt(m[1]*m[1] + m[5] * m[5] + m[9] * m[9]);
scale.z = mSqrt(m[2]*m[2] + m[6] * m[6] + m[10] * m[10]);
return scale;
}

#11 This fix works in my test cases.

#12 Thanks. :)

#13 Looking at TGEA 1.7, this fix seems to be in there (or at least something similar in ProcessedShaderMaterial::setEyePosition). However, I'm having some issues that seem related to this. I have an atmosphere shader, and it works fine with TGEA 1.0.3. However, with TGEA 1.7, the shader behaves strangely.

This pic shows the shader working correctly. The planet and the atmos'sphere' are scaled 1 1 1.

Now, if I scale things, the atmosphere gets screwed. This pic has the planet scaled 5 5 5, and you can clearly see the messed up atmosphere rendering.

Now, if I comment out the position.convolveInverse(objTrans.getScale()); line in ProcessedShaderMaterial::setEyePosition, the shader renders as it should, irrespective of the objects scale, like this one which is scaled 3 3 3.

The shader is pretty simple - it simply modulates the alpha based on both the angle to the light source and the angle to the viewer. All the vectors are normalized.

Do you think this a side-effect of the fog fix??

#14 The fog calculation needs the position of the eye in a "funny" coordinate system. It wants it in the object space of the object being rendered, but with world units for measurement so it can access the fog texture correctly.

So, if you use the eye position to calculate a vector and normalize it before doing anything else with it, this change shouldn't have affected you. But if you do something with the result and then normalize it, it would change the behavior.

It's hard to say what's going on here without seeing the shader. But I'd wager something is using eyePos before it's normalized.

#15 So does that mean that the eye position is being given using a different transformation than everything else like pos, normal etc?

I'm only using eyePos one place in the shader, and the line reads:
float3 viewDir = normalize(In.pos - In.eyePos);

Now, I guess that line assumes that both pos and eyePos are in the same coordinate system. So if pos is in the object coordinate system that includes scale, and eyePos is the objects coordinate system that has the scale removed (by the objTrans.convolveInverse line) then viewDir as calculated above isn't actually going to give the vector we're expecting.

Here's an example:
An object that is unscaled has scale {1 1 1}
If we take pos to be {0 0 1}
and eyePos to be {0 1 2}
then pos-eyePos = {0 -1 -1}

Now if we scale the object {2 2 2}
pos = {0 0 2}
eyePos = {0 2 4}
pos-eyePos = {0 -2 -2}
which is equivalent to the vector for the unscaled object - this is how TGEA 1.0.3 did things.

now, if we take the TGEA 1.7 route and leave pos unchanged but inverseConvolve the eyePos by the objects scale,
pos = {0 0 2}
eyePos = {0 1 2}
pos-eyePos = {0 -1 0} - YUK!

Is that logic correct??

If so, then I guess we could do with two shaderConsts for the eyePosition - one with scale information, and one without.

#16 The IN.pos should be pre-scaled as well, because scale is applied with the modelview matrix. So the units should be matching.

But two shader constants would be a solution as well. It'd be easy for you to add that to your tree.

#17 I don't quite follow -
on line 668 of processedShaderMaterial.cc the scale is being removed from VC_EYE_POS only.
I can't see this being done for the transforms of anything else - they seem to be passed with the scale intact giving two different coordinate systems.

#18 IN.position is the raw vertex data coming from the DTS shape. It isn't scaled on the CPU side. If it were, we'd need to have a separate set of vertex data for each instance of that shape. But we don't. The scale factor is applied with this line in the shader:

OUT.hpos = mul(modelview, IN.position);

If you are using IN.position, then that's the unscaled/untranslated/unrotated object space coordinate. Which should mean that the units are the same. But I maybe missing a detail that makes this all wrong. Hope you find a solution!