Bug fix for shadow rendering of tri-list primitives

I've come a across and bug in realtime shadow rendering when the mesh that the shadow is rendered from contains TSDrawPrimitive::Triangles as well as TSDrawPrimitive::Strip type primitives. This will happen if you set $pref::TS::UseTriangles = 1 in your prefs, or if you you change TSMesh::smMinStripSize to anything other than 1 (at the top of TSMesh.cc). The issue that caused it to appear for me was the later, as I found that I was getting better performance on my Radeon X1600 with TSMesh::smMinStripSize = 10 (so that I wasn't having very small tristrips, like 1 or 2 polys big).

Anyways, there are 2 bugs with the call to TSMesh::renderShadow(), and I'm posting a fix to both of them below.

Ok, issue #1 is that TSMesh::renderShadow() assumes that all primitives that make up the mesh will be the same (ie. either in strip or list format, not both), which is an obvious issue. In order to fix this, you need to replace:

// pick correct bit render routine...we assume all strips or all triangles
   if ( (primitives[0].matIndex & TSDrawPrimitive::TypeMask) == TSDrawPrimitive::Strip)
      BitRender::render_strips((U8*)primitives.address(),primitives.size(),sizeof(TSDrawPrimitive),indices.address(),gShadowVerts.address(),dim,bits);
   else if ( (primitives[0].matIndex & TSDrawPrimitive::TypeMask) == TSDrawPrimitive::Triangles)
      BitRender::render_tris((U8*)primitives.address(),primitives.size(),sizeof(TSDrawPrimitive),indices.address(),gShadowVerts.address(),dim,bits);
   else
      AssertFatal(0,"TSMesh::renderShadow: strips or triangles only...how'd you get in here.");

with

for(U32 i=0;i<primitives.size();i++)
	{
		if((primitives[i].matIndex & TSDrawPrimitive::TypeMask) == TSDrawPrimitive::Strip)
		{
			BitRender::render_strips((U8 *)primitives.address() + (i * sizeof(TSDrawPrimitive)),  
										1, 
										sizeof(TSDrawPrimitive), 
										indices.address(), 
										gShadowVerts.address(), 
										dim, 
										bits);
		}
		else if((primitives[i].matIndex & TSDrawPrimitive::TypeMask) == TSDrawPrimitive::Triangles)
		{
			BitRender::render_tris((U8 *)primitives.address() + (i * sizeof(TSDrawPrimitive)), 
									1, 
									sizeof(TSDrawPrimitive), 
									indices.address(), 
									gShadowVerts.address(), 
									dim, 
									bits);
		}
   }

This will cause each individual primitive list to be sent through for shadow rendering based on its type (ie. strip or list).


Issue #2 is that BitRender::render_tris() is actually broken in 2 places. I assume that this has never come up for anyone as Torque always does full tristripping by default...

Anyways, the fixes are:

-Around line 327 (look for "(v1->y==v2->y)"), there are 4 variables that need to be initialized but are not, so you need to add:

xLeftInc = xRightInc = 0;
xLeftDir = xRightDir = 1;

I took this from BitRender::render_strips() as it is done there, but not in render_tris() for some reason...


-Secondly, around 272 where the backface culling check is done (look for "((v0->x-v1->x)*(v2->y-v1->y) > (v0->y-v1->y)*(v2->x-v1->x))"), the function returns instead of just continuing to the following triangle in the list, so change:

return;

continue;

What I've actually done on my side, is taken out the inner sections of both BitRender::render_tris() and BitRender::render_strips() as they are IDENTICAL, and just make each function call this single function.