Cel Shading in Torque - An Evolutionary Approach

#1 The post length limit forced me to move some of my comments here.

Here's a screen shot of "flat cel shading":


Since the light hits the player model in the face, you'll need to use camera mode (or have another player log in) to see the shading.

Also, this limits cel shading to objects, not buildings or terrain--which is what I wanted. It shouldn't be too hard to transfer the concept to the other render methods, though.

In a nutshell, what this does is cause OpenGL to add the shade color like an alpha value, making the polygon texture brighter or darker.

Unfortunately, besides being flat shaded, this method has issues with reflective environment mapping on objects (like the orc in RW). I'm open to suggestions for how to work past that.

I went with a hard-coded light vector, rather than tracking the sun because that requires the object transform, and getting the object transform into the render code is its own little PITA (unless I've missed something). I'll talk about that more in a later post.

I'll post another example of a method I tried either later today or tomorrow. Comments and critiques are welcome.

-David

Edit - Added a bit more explanation.

#2 The Second Attempt - Smooth-Flat Cel Shading

In an attempt to smooth out the shading when it's "in motion" (which is to say "in a game environment and not just looking cool on a screenie"), I decided to set the shade color at each vertex and then let OpenGL interpollate the shades across the poly.

Since the render method uses glDrawElements() with glVertexPointer() et al, that meant I had to create an array of color values, one per vertex. The color array is built using the dot product of the individual normal with the light vector, and then passed to OpenGL.

Here's the new TSMesh::render():

void TSMesh::render(S32 frame, S32 matFrame, TSMaterialList * materials, const MatrixF *objectTransform)
{

   ...

   const Point3F * normals = getNormals(firstVert);
   saveMergeNormals(); // verts & tverts saved and restored on tsshapeinstance::setStatics

   [b]// drm - cel shading setup
   VectorF lightVector(1,1,1);
   Vector<Point4F> colors;
   colors.setSize(vertsPerFrame);
   for (S32 vv=0; vv<vertsPerFrame; vv++)
   {
      Point3F tmpNormal=normals[vv];
      // dot product of the vertex normal and light
      F32 dotP = mDot(tmpNormal,lightVector);
      // adjust to cel shading
      F32 shade;
      if (dotP<=-0.50)
         shade=0.35;
      else if (dotP<=0.00) 
         shade=0.45;
      else if (dotP<=0.45)
         shade=0.65;
      else
         shade=1.0;
      colors[vv]=Point4F(shade,shade,shade,1.0);
   }
   glEnableClientState ( GL_COLOR_ARRAY );
   glColorPointer(4,GL_FLOAT,0,colors.address());
   // [/b]
   ...

   // lock...
   bool lockArrays = dglDoesSupportCompiledVertexArray();
   if (lockArrays)
      glLockArraysEXT(0,vertsPerFrame);

   [b]// drm - disable lighting
   oldlighting=glIsEnabled(GL_LIGHTING);
   glDisable(GL_LIGHTING);[/b]

   for (S32 i=0; i<primitives.size(); i++)
   {
      ...
   }

   [b]// drm - finish up
   if (oldlighting) glEnable(GL_LIGHTING);
   glDisableClientState ( GL_COLOR_ARRAY );[/b]

   // unlock...
   if (lockArrays)
      glUnlockArraysEXT();

   restoreMergeNormals();
}

The result is "kinda cel shading". An interesting shading model, I think, sort half way between the flat cel shading and the usual shading done by OpenGL. It's "brighter" than the usual OpenGL shading, because more of the front face of the model is fully light, with a rapid falloff in luminence to the sides and back.

The kicker, though, if you want real cel shading, is that the line of the shading is soft and fuzzy. It's not that gorgeous hard edge that makes us want cel shading in the first place.

NOTE: This method also has the problem with reflective environment maps, BTW, so keep that in mind.

The next method I tried, using multitexturing, actually does what I want. But I'm still working some kinks out (environment mapping seems to taint all it touches), so it might be tomorrow or this weekend before I can provide a simplified explanation of it.

Again, comments and critique are welcome.

-David

#3 Awesome. i'll have to check it out later.

#4 I wouldn't worry too much about environment mapping, I doubt it would look good with cel-shading anyways and shouldn't be used on any cel-shaded stuff. Though I can imagine if you are looking at multi-texturing it might be screwing with that in general. I wonder if you could use the lighting pack as reference for getting the light data, as it has much more advanced lighting for doing dynamic shadows and such, so it has to be using a dynamic setup for getting it's light direction for shadows... I'd think anyways.

#5 @David,

This looks really great!. I tried to bring this in to my latest build, but I get one error:

"tsMesh.cc(278): error C2065: 'lightVector' : undeclared identifier"

and this is line 278:

"F32 dotP = mDot(tmpNormal,lightVector);"

I have the "VectorF lightVector(1,1,1);" further up, but what I'm thinking, considering I have TLK and have integrated the DRL resource, that somehow they're part of the problem.

- Alan
Really Really Good Things Studio

#6 Paul,

I agree about env mapping in a cel shaded world. Just wanted to mention it *before* someone ran with starter.fps and saw the orc in his spiffy new sky-reflecting pajamas. ;-)

I have the lighting kit on order (had to mail it in because some company I won't mention doesn't accept Amex) and I'm really looking forward to seeing what it can do for me.

-David

#7 Alan,

You might make sure that the lightVector declaration isn't inside a nested code block. I've done that to myself a few times in the course of this.

Let me know what you find out when you get it working.

-David

#8 A little Doh!! is apparently appropriate. Compiles now, no errors but no cell shading effect either. I'm sure I've hosed something. I'll keep messing with it until I'm sure I have to restore one of my backups...=)

Edit: Ok, I think it's working, but the oversexposure of the DRL resource and TLK combined wash out the character/player and dts objects a lot. I'll keep messing with it. Thanks, David.

- Alan

#9 Clint implemented a cell-shading technique which basically environment-maps the character with a cube map that has the toon-shading in it. it looks great.

#10 The first screen shot look really good.

#11 Alan: If you haven't disabled the lighting, OpenGL will override you. That's about all I can think of.

Orion: We experimented with the env map approach (or my artist did). We decided to try it in code, and go for the full-on cel shading. So far, we're pretty pleased with what we've come up with--which is almost presentable.

-David

#12 Ditto on the presentable.
i think we'll be able to share screenshots in the next couple weeks or so.

#13 The Third Attempt - Honest-to-God Cel-Shading in Torque

OK, this one requires a bit more work, and more explanation. As such, it will probably span at least 2 posts.

This method is based on multi-texturing and uses 2 passes. The resulting look, though, seems to be well worth it. And since you can limit which objects get the cel-shading (players, statics, etc) you don't have to take the 2-pass performance hit on everything. Also, I'm only interested in cel shading objects (DTS) not interiors or the terrain. Still, you could probably extrapolate how to cel shade those, as well, if you wanted.

NOTE: This method doesn't require (and in fact really won't work with) the early 2 methods. So start with clean 1.4.

REPEAT: Don't use the code I posted in methods #1 & #2 with this one.

EDIT: Since I can't keep this up-to-date, I'm removing the code changes from this thread. You can find all of the updated code changes here:

Cel Shading in TGE


What I want to change/extend are:
* Work with interior lights (still waiting to get the lighting kit)
* Have a scriptable, per object cel shading texture
* Better integrate with Torque's multitexturing so I can get it all done in one pass

Known issues:
* Statics with translucent materials (like the trees in the starter.fps) show some artifacts as the cel shading is sometimes appears on top of the billboard thingy.
* Player models with environment maps sometimes show artifacts of the cel shading of other player models in the scene. Still looking for the source of this problem. If you find it, let me know. :)

Comments and critique--and suggestions for improvements--are welcome.

Have fun with it.

-David

#14 Farking sweet! but wow, that's alot of changes, perhaps you should go ahead and make a resource for this, with the cc files, and then give me a link to it right away so i can play with it :)

#15 I posted it as a resource.

Cel Shading in TGE

Dunno how long it will take to get moderated.

Here's the resource file, with all of the above:
Cel Shading in TGE (zip file)

-David

Edit: Added file link.

#16 David, thanks so much for posting this. I don't need it for my current projects, but this is just more proof of how awsome this community is. Excellent.

#17 @David,

You knew I'd be back, didn't you? =)

Ok, so here's where I'm at. I used the third attempt from the thread above (using clean TS files, from a saved build). I kept getting two errors over and over again. (I'm sorry I don't remember what they were right now, because...well, the following will probably explain).

So I figured I couldn't cut and paste worth beans, so I noticed only one reference to TLK, between the files in the zip resource and mine, so I just dropped them into the ts directory and overwrote the current files. This time 28 build errors, so I say to my self, it's probably one of the gazillion changes I've made over the last 14 months, so I go to a clean 1.4 directory, drop them in...and get the same errors. I'm at a loss at what I'm doing wrong. I integrated TLK 1.3.5 beta by hand, and these few changes are stumping me. I feel like and idiot. Anyway, here are the errors I'm getting. And thanks again for your help.


C:\Torque\engine\ts\tsShapeInstance.cc(1275): error C2039: 'renderCelShade' : is not a member of 'TSShapeInstance'
C:\Torque\engine\ts\tsMesh.cc(1343): error C2065: 'gClientSceneGraph' : undeclared identifier
C:\Torque\engine\ts\tsShapeInstance.cc(1291): error C2065: 'mMaterialList' : undeclared identifier
C:\Torque\engine\ts\tsShapeInstance.cc(1291): error C2065: 'mMeshObjects' : undeclared identifier
C:\Torque\engine\ts\tsShapeInstance.cc(1278): error C2065: 'mShape' : undeclared identifier
C:\Torque\engine\ts\tsShapeInstance.cc(1277): error C2065: 'smRenderData' : undeclared identifier
C:\Torque\engine\ts\tsShapeInstance.cc(1278): error C2227: left of '->details' must point to class/struct/union
type is ''unknown-type''
C:\Torque\engine\ts\tsMesh.cc(1343): error C2227: left of '->getLightManager' must point to class/struct/union
type is ''unknown-type''
C:\Torque\engine\ts\tsShapeInstance.cc(1282): error C2227: left of '->subShapeFirstObject' must point to class/struct/union
type is ''unknown-type''
C:\Torque\engine\ts\tsShapeInstance.cc(1283): error C2227: left of '->subShapeNumObjects' must point to class/struct/union
type is ''unknown-type''
C:\Torque\engine\ts\tsShapeInstance.cc(1289): error C2228: left of '.currentTransform' must have class/struct/union type
type is ''unknown-type''
C:\Torque\engine\ts\tsShapeInstance.cc(1294): error C2228: left of '.currentTransform' must have class/struct/union type
type is ''unknown-type''
C:\Torque\engine\ts\tsShapeInstance.cc(1277): error C2228: left of '.detailLevel' must have class/struct/union type
type is ''unknown-type''
C:\Torque\engine\ts\tsShapeInstance.cc(1291): error C2228: left of '.renderCelShade' must have class/struct/union type
C:\Torque\engine\ts\tsShapeInstance.cc(1276): error C2365: 'renderCelShade' : redefinition; previous definition was a 'formerly unknown identifier'
C:\Torque\engine\ts\tsShapeInstance.cc(1608): error C2664: 'void TSShapeInstance::render(const MatrixF *,const Point3F *)' : cannot convert parameter 1 from 'S32' to 'const MatrixF *'
Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast
C:\Torque\engine\ts\tsShapeInstance.cc(1616): error C2664: 'void TSShapeInstance::render(const MatrixF *,const Point3F *)' : cannot convert parameter 1 from 'S32' to 'const MatrixF *'
Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast
C:\Torque\engine\ts\tsShapeInstance.cc(1282): error C3861: 'mShape': identifier not found, even with argument-dependent lookup
C:\Torque\engine\ts\tsShapeInstance.cc(1283): error C3861: 'mShape': identifier not found, even with argument-dependent lookup
C:\Torque\engine\ts\tsShapeInstance.cc(697): error C3861: 'renderCelShade': identifier not found, even with argument-dependent lookup
C:\Torque\engine\ts\tsShapeInstance.cc(1289): error C3861: 'smRenderData': identifier not found, even with argument-dependent lookup
C:\Torque\engine\ts\tsShapeInstance.cc(1294): error C3861: 'smRenderData': identifier not found, even with argument-dependent lookup

- Alan

#18 It sounds like you didn't update the header files properly? Just based on all the undeclared identifiers...

#19 My bad. I did leave out an include in TSMesh.cc (scenegraph.h). I updated the post (pt 1) and the zip file. I'll update the resource now.

EDIT - And it seems I forgot to add one of the function headers in TSShapeInstance.h. That's what happened to you, Alan. My mistake. File's been updated.

-David

#20 @David,

My wife paid you to make me insane...didn't she... =P

No problem at all, thanks again for a *TON* of hard work,

- Alan

Edit: BTW, I'm back to the same two original errors...LOL!

C:\Torque1.4L Monster Island\engine\ts\tsShapeInstance.cc(1608): error C2664: 'void TSShapeInstance::render(const MatrixF *,const Point3F *)' : cannot convert parameter 1 from 'S32' to 'const MatrixF *'
C:\Torque1.4L Monster Island\engine\ts\tsShapeInstance.cc(1616): error C2664: 'void TSShapeInstance::render(const MatrixF *,const Point3F *)' : cannot convert parameter 1 from 'S32' to 'const MatrixF *'
Conversion from integral type to pointer type requires reinterpret_cast, C-style cast or function-style cast