Get an 20% OFF using the RELEASE code on your book purchase. For a limited time.

Introduction to the shader programming language

3.2.4. ShaderLab Stencil

This post is also available in…

According to the official documentation in Unity: 

“The Stencil Buffer stores an integer value of eight bits (0 to 255) for each pixel in the Frame Buffer. Before running the fragment shader stage for a given pixel, the GPU can compare the current value in the Stencil Buffer with a determined reference value. This process is called Stencil Test. If the Stencil Test passes, the GPU performs the depth test. If the Stencil Test fails, the GPU skips the rest of the processing for that pixel. This means that you can use the Stencil Buffer as a mask to tell the GPU which pixels to draw and which pixels to discard”.

What does this mean? To understand it, we must take into consideration that the Stencil Buffer itself, is a “texture” that must be created and for this, it stores an integer value from 0 to 255 for each pixel in the Frame Buffer.

(Fig. 034)

As we already know, when we position objects in our scene, their information is sent to the vertex shader stage (e.g. vertex position). Within this stage, the attributes of our object are transformed from object-space to world-space, then view-space, and finally clip-space. This process occurs only for those objects that are inside the frustum of the camera. 

When this information has been processed correctly, it is sent to the rasterizer which allows us to project in pixels the coordinates of our objects in the scene, however, before reaching this point it goes through a previous stage of processing called Culling and Depth testing. In this stage various processes occur that we can manipulate within our shader, among them are Cull, ZWrite, ZTest and Stencil.

Basically what the Stencil Buffer does is activate the Stencil Test, which allows the discarding of “fragments” (pixels) so that they are not processed in the fragment shader stage, thus generating a mask effect in our shader. The function performed by the Stencil Test has the following syntax:

if ( StencilRef & StencilReadMask [Comp] StencilBufferValue & StencilReadMask)
{
    Accept Pixel.
}
else
{
    Discard Pixel.
}

StencilRef” is the value that we are going to pass to the Stencil Buffer as a reference, what does this mean? Remember that the Stencil Buffer is a “texture” that covers the area in the object’s pixels. The StencilRef works as an id that maps all the pixels found in the Stencil Buffer.

For example, we are going to make the StencilRef value 2.

(Fig. 035)

In the example above, all the pixels covering the capsule area have been marked with the value of the StencilRef, therefore now the Stencil Buffer equals 2. 

Subsequently, a mask (StencilReadMask) is created for all those pixels that have a reference value that by default has the value 255.

(Fig. 036)

Thus, the above operation is as follows. 

if ( 2 & 255 [Comp] 2 & 255)
{
    Accept Pixel.
}
else
{
    Discard Pixel.
}

Comp” refers to a comparison function that gives a true or false value. If the value is true, the program writes the pixel in the Frame Buffer, otherwise, the pixel is discarded. 

Within the comparison functions, we can find the following predefined operators:

OperatorFunction
Comp NeverThe operation will always deliver false. 
Comp Less<.
Comp Equal==.
Comp LEqual≤.
Comp Greater>.
Comp NotEqual!=.
Comp GEqual≥.
Comp AlwaysThe operation will always deliver true.

We need at least two shaders to use the Stencil Buffer: One for the mask and one for the masked object. 

Let’s suppose we have three objects in our scene: A cube, a sphere and a square. The sphere is inside the cube, and we want to use the square as a “mask” to hide the cube so that the sphere inside can be seen. To create this example mask, we will create a shader called USB_stencil_ref and its syntax would be the following:

SubShader     
{
    Tags { “Queue” = “Geometry -1” }
    ZWrite Off
    ColorMask 0

    Stencil
    {
        Ref 1
        Comp Always
        Pass Replace
    }
}
(Fig. 037)

Let’s analyze the above. The first thing we did was configure “Queue” equals “Geometry minus one”. Geometry defaults to 2000, therefore, Geometry minus 1 equals 1999, this will process our square (mask), to which we will apply this shader, first in the Z-Buffer.

However, as we know, Unity by default processes objects according to their position in the scene concerning the camera, therefore if we want to disable this function we must set the property “ZWrite to Off”. 

Then we set “ColorMask to zero” so that the mask pixels are discarded in the Frame Buffer and appear transparent. 

At this point our square still does not work as a mask therefore what we must do next is add the command “Stencil” so that it functions as such. 

“Ref 2” (StencilRef), our reference value, is compared on the GPU with the current content of the Stencil Buffer using the operation defined in the “comparison operation”. 

“Comp Always” makes sure to set a “2” in the Stencil Buffer, taking into account the area covered by the square on the screen. Finally “Pass Replace” specifies that the current values ​​of the Stencil Buffer be “replaced” by the StencilRef values.

Now we are going to analyze the object that we want to mask. We will create another shader called USB_stencil_value to illustrate this. Its syntax is as follows:

SubShader 
{
    Tags { “Queue” = “Geometry” }
    Stencil
    {
        Ref 1
        Comp NotEqual
        Pass Keep
    }
}
(Fig. 038)

Unlike our previous shader, we will keep the Z-Buffer active so that it is rendered on the GPU according to its position relative to the camera. Then we add the Stencil command so that our object can be masked. We add “Ref 2” again to link this shader with USB_stencil_ref for the mask. 

Then, in the comparison operation, we assign “Comp NotEqual”. This states that the cube area exposed around the square will be rendered because the Stencil Test passes (no comparison or true). 

On the other hand, for the square area, being equal (Equal) the Stencil Test will not pass, and the pixels will be discarded. “Pass Keep” means that the Cube maintains the current content of the Stencil Buffer. 

If we want to work with more than one mask, we can give a Ref property number different to “2”.

Follow us to stay informed about all the latest news, updates, and more.

Join the group to share your experiences with other developers.

Subscribe to our channel and keep learning game dev!

jettelly-logo

Jettelly Team

We are a team of indie developers with more than 9 years of experience in video games. As an independent studio, we have developed Nom Noms in which we published with Hyperbeard in 2019. We are currently developing The Unity Shader Bible.

Follow us on our social networks.