Alpha Masking

#1 I couldn't find any support for masks like there is in Flash (obj.mask = otherObj) but thankfully I only need support for unrotated rectangles.

This was fairly easy to implement in a shader but I had to modify the engine source pretty heavily to add it as a basic T2DSceneObject feature.

Coming from the Flash world I've been too pampered with their ultra user friendly functions :P

If anyone cares for the details drop me a line.

#2 Sounds interesting. One approach would have been to make a big screen sized sprite with a hole in it and make sure anything that needs cliipping is behind that and anything that doesn't is infront. Although it would depend on your precise needs.

In Direct X (and thus xna) the term for what you are talking about is 'stencilling' and is implemented using stencil buffers - unfortunately Torque X doesn't come with that feature automatically supported as far as I know.

I'm certainly interested to hear about your shader solution though.

#3 Stencilling you say? Excellent, I was really stuck on what the term was. Maybe I'll look into that one day and see if I can put something together more like how Flash works :)

The giant hole sprite does work for a lot of circumstances and I thought about using it but eventually got scared off by the complexity that would ensue. I have multiple scenarios where I would have to do a lot of layer management as well as making doubles of moving backgrounds.

One such scenario looks like the following (please excuse my terrible programmer art :))

The background has clouds that scroll across it randomly using SceneObjects. Now, I could technically just make an animated sprite with a big hole in that looks like moving clouds, but it would be one of those memory, dynamic-ness, hard to change, etc trade-offs.

Each of the four rectangles in the pic are playing areas where objects need to be cropped to. The objects can fly out of any side of the fields and I don't want them to overlap onto the other fields. I want them cropped. For this situation I couldn't think of a layering scheme to prevent this. There may be one, but this is just one of a few similar scenarios where things like this can happen and I didn't want to solve that many problems :S


So, I decided on a shader. Luckily I only deal with rectangle areas that aren't rotated so it was fairly easy once I got a handle on what a shader represents. Basically, what I did was the following:

1. Added T2DSceneObject _mask; to the T2DSceneObject class, allowing me to assign any scene object as a mask

2. Modifying all of the RenderInstance classes, I allowed them to accept TopLeft, BottomRight and HasMask parameters.

The TopLeft parameter was a Vector2 which represented the Mask's top left corner in screen space. So between (-1,-1 BL of screen) and (1,1 TR of screen).

The BottomRight was similar, but represented the bottom right.

Finally HasMask was whether or not this RenderInstance was masked.

3. I passed these parameters to a shader which then checked the current pixel it was drawing against the bottomright and topleft parameters. If it was contained within these constraints, it drew the pixel as normal. However, if it was not contained, it set the alpha to 0 so that it was not visible.


The relevant shader code is here:

float2 maskTopLeft; // ADDED BY ALEX
float2 maskBottomRight; // ADDED BY ALEX
bool hasMask; // ADDED BY ALEX

struct VSOutput
{
...	
	float3 maskScreenPosition : POSITION1;	
};

VSOutput SimpleVS(VSInput input)
{
	VSOutput output;
	output.position = mul(input.position, worldViewProjection);
	.....
        output.maskScreenPosition = output.position; // ADDED BY ALEX
}

float4 SimplePS(VSOutput input, uniform bool useColor, uniform bool useTexture) : COLOR
{
....
	if (hasMask)
	{
		if (input.maskScreenPosition.y > maskTopLeft.y || input.maskScreenPosition.y < maskBottomRight.y  || input.maskScreenPosition.x < maskTopLeft.x || input.maskScreenPosition.x > maskBottomRight.x)
		{	 			
			color.a = 0; // Can't see
		} 				
	}
    return color;
}

And to calculate the topleft and bottomright based off a T2DSceneObject is here:

// Quick constants
            float objW2 = Mask.Size.X / 2.0f;
            float objH2 = Mask.Size.Y / 2.0f;
            float screenWidth2 = Game.Instance.GraphicsDevice.PresentationParameters.BackBufferWidth / 2.0f;
            float screenHeight2 = Game.Instance.GraphicsDevice.PresentationParameters.BackBufferHeight / 2.0f;
         
            // Formulas to calculate the box in screen space
            float topLeftX = (Mask.Position.X - objW2) / screenWidth2;
            float topLeftY = (Mask.Position.Y - objH2) / -screenHeight2;
            float bottomRightX = (Mask.Position.X + objW2) / screenWidth2;
            float bottomRightY = (Mask.Position.Y + objH2) / -screenHeight2;

            // Set the info
            _sceneObject.MaskTopLeft = new Vector2(topLeftX, topLeftY);
            _sceneObject.MaskBottomRight = new Vector2(bottomRightX, bottomRightY);
            _sceneObject.HasMask = true;

I would put more source code but I modified a lot of GarageGames engine classes and I don't want to get in trouble for revealing their source.

If you have any questions or suggestions just let me know. I'm very new to shaders

#4 My only concern (and it's a very small one) would be if step 3 means that other shaders can't be used unless modified to use the needed parameters. Although that's a pretty small concern really as it would take very little effort to mod a shader to do the masking.

#5 An alternative approach might be to render the set of objects, that need clipping, onto a temporary texture that itself is then rendered to the final scene. The rendering to texture would do rectangular clipping and in rendering the final scene a further shader can be used that can take an alpha mask if a hard rectangle is not what is required.

Although proper stencilling would probably be better obviously :) Just throwing out another idea to add into the mix.

#6 Originally I was thinking about rendering to a texture but I couldn't figure out how to do it. Basically I wanted to create a texture the size of the object that is being masked. Then transform the mask into the object's space and render to the texture. That texture would then be used as an alpha map for the shader. Any parts that weren't black would be rendered normally.

Something like this:
1. Scene objects in game in world space

2. Object + Mask transformed to Object Space

3. Combined alpha map texture

Then in a shader comparing the object texture to the alpha map texture and any non-black parts get rendered.

Unfortunately I have no idea how to set up a texture in a certain location in the world and have objects drawn to it. Essentially, cropping out a section of the game world eludes me.


You raise a good point regarding multiple objects using the same mask. This method would be pretty inefficient since lots of textures would be created (one for each mask). It would likely be better to do this in the other way, and draw all masked objects into the mask's space. Then when drawing you could sample the alpha texture with some translation logic and have the renderer worry about what draw on top of what.

In any case, I have no idea how to do any of that with shaders so I'll probably just look into stencilling when I need it. Although, it might be a good exercise

#7 Yes, rendering the objects onto a single target texture was what I had in mind, rather than the other way around.

The simplest approach is probably rendering the objects to an intermediate texture the size of the screen (so no worries about translating object coordinates) and then that texture is passed through a masking shader that applies a mask. (if many separate masks were being used there might be perfomance issues obviously, since each mask would require rendering a screen sized intermediate texture - in which case optimized smaller intermediate textures would become more useful).

Stencilling requires modifying the depth buffer and various tweaks like that. I've not messed with that sort of thing in the engine yet so I don't know how easy it would be to integrate such a feature.

#8 Could this have been done using two scene views or two cameras? For instance, in some of the examples, they have mini-maps on top of the main play area. Your background with the clouds could be one scene/camera and then your second camera would be put over the original for your play area...know what I mean?

#9 I hadn't thought of that - which examples are those?

I must admit I haven't played around with multiple scene views. Although I do load multiple scenes they 'merge' in the same 'view' rather than in separate ones.

#10 Hey Jake, I can see that working if you make multiple render passes in the draw section (one for each camera) and draw each one to a separate texture. Then position the textures on top of each other for the final scene. I don't think there's default support for this though, but it would be something nice to have for HUDs like you mention. Especially for 3D games where you want to show what's happening in a different part of the level.

Not sure if it's the same example but the only HUD example I saw with a mini map manually drew circles (or squares) where the enemies were, based on some math. They didn't draw them if they were out of bounds :( For that example they didn't clip at all.


Anyhow, for this thread I want to stay on generic alpha masking/compositing/stencilling. So, for example if each of the play areas was a Star, or animated running sprite, the mask should only display those areas.

Your idea with multiple cameras could probably solve this by rendering the mask as a huge black texture with a hole in it for whichever camera contained the masked objects. So, you would render the whole scene of masked objects, then render the giant black mask on top of it (layer 0 or something). Afterwards, when combining the two layers, you could crop out all the black stuff.... although, this might result in any pure black being used being cut out =/ Just a random thought

#11 I'm not sure, but it might be possible to have multiple SceneViews at once (and I guess they would render with different cameras automatically) - I can't remember off the top of my head though as I haven't messed around with that before. Perhaps someone else will know?

#12 Update: I can confirm that it's possible to have multiple sceneviews of arbitrary size and position and have them clip their content. Some minor engine modding legwork is required.

It's rectangles only obviously unless you add a foreground object to act as a mask, but thought I'd update with latest findings incase anyone thinks it might be useful to them.