Stencil Help

[Alp]

I'm trying to create a masking effect for a character select screen, and I can't seem to make heads or tails of how stencils work.

It really doesn't help, that the only examples I found on the wiki, seem to be strictly vector-based. Is it even possible to mask an image, using another image, as a guide? If so, could somebody please post an example? I'm quite stumped.

Since my art is NSFW, I'll post a simplified diagram, showing what I'm going for:


The basic idea is that there's a spotlight in the middle of the screen, and the character sprites are displayed as negative outlines outside of the spotlight, and fully colored art, inside the spotlight.

[Ref]

Maybe something like this in reverse?
(Very old code so don't know how good it is)

[pgimeno]

Welcome to the forums. By default, alpha is not taken into account when drawing to the stencil, only the geometry. That can be changed with a shader. For example, this shader makes every pixel with alpha < 0.5 to not be drawn to the stencil:

Code: Select all

vec4 effect(vec4 colour, Image texture, vec2 texpos, vec2 scrpos)
{
    vec4 pixel = Texel(texture, texpos) * colour;
    if (pixel.a < 0.5) discard;
    return pixel;
}

See attachment for an example using your graphics.

[Alp]

Maybe something like this in reverse?
(Very old code so don't know how good it is)

Oh, that's a neat effect! It's not quite what I'm going for (a bit too noisy for my liking), but it looks great!
My art is strictly pixel-based, so I want really sharp lines.

Welcome to the forums. By default, alpha is not taken into account when drawing to the stencil, only the geometry. That can be changed with a shader. For example, this shader makes every pixel with alpha < 0.5 to not be drawn to the stencil:

Code: Select all

vec4 effect(vec4 colour, Image texture, vec2 texpos, vec2 scrpos)
{
    vec4 pixel = Texel(texture, texpos) * colour;
    if (pixel.a < 0.5) discard;
    return pixel;
}

See attachment for an example using your graphics.

Ah! Thank you! That's EXACTLY what I was going for! So, you can use the alpha for stencils? Huh, good to know! I was under the assumption that stencils were strictly bitmask-based.

So, "pixel.a" is the value of the alpha channel for a given pixel, correct? Just making sure I understand what the code is doing.

For what it's worth, attempts were made, based on information from the wiki, but the closest I was able to get, was inverting the bit mask (black to white, and white to black), and I don't even understand how THAT worked!

I've attached my previous attempts, and the result, based on the code you provided. (They're NSFW, so fair warning.)

test_stencil2.love

NSFW

(15.05 KiB) Downloaded 295 times

test_stencil3.love

NSFW

(14.83 KiB) Downloaded 297 times

spotlight10.love

(64.28 KiB) Downloaded 293 times

"stencil2 + stencil3" are my attempts, but I couldn't quite get it right, after a few hours of struggling with it.
"spotlight10" is the result I was aiming for, based on your example. The curtains are a placeholder from Mario 2, for the moment.

Could the example code you provided be posted to the wiki? It would be VERY helpful for others, I imagine!

EDIT: The forums dropped my attachments, argh.

[pgimeno]

Cool, and nice graphics!

Ah! Thank you! That's EXACTLY what I was going for! So, you can use the alpha for stencils? Huh, good to know! I was under the assumption that stencils were strictly bitmask-based.

A stencil is a buffer that holds numbers. It's not a bitmask, but you can apply a comparison method to it and the result is always a boolean which tells whether each corresponding pixel should be drawn or ignored. By default the stencil function sets it to 1, so I used "equal to 1" or "not equal to 1" as comparison functions.

So, "pixel.a" is the value of the alpha channel for a given pixel, correct? Just making sure I understand what the code is doing.

Yes, in GLSL you can address the elements of a vec4 with .xyzw or with .rgba (.x is the same as .r and so on).

If you don't know about shaders, there's a good introduction to the basics in the LÖVE blog: http://blogs.love2d.org/content/beginners-guide-shaders and the default shader used by LÖVE (very similar to the one I used) is here: https://love2d.org/wiki/love.graphics.newShader#Notes_2

Could the example code you provided be posted to the wiki? It would be VERY helpful for others, I imagine!

There's one example already which uses black for transparent instead of an alpha threshold: https://www.love2d.org/wiki/love.graphi ... encil_mask