How to use a transparent image/canvas as a stencil

[Bobble68]

I am once again humbly throwing myself before the almighty forum of LÖVE for help.

I'm trying to draw a canvas with transparency that I can use as a stencil elsewhere in the program, however I've found that the stencil just uses the whole canvas as a rectangular stencil, rather than just the parts which are filled in. I tried doing the same thing with a test image, and found the same thing happening.

Is there a way of doing this? My guess would be to stencil based on the alpha value of the pixel, but I'm unsure how I would do that.

[darkfrei]

One example
https://github.com/darkfrei/love2d-lua- ... in/stencil

[Bobble68]

One example
https://github.com/darkfrei/love2d-lua- ... in/stencil

How does this work exactly? I haven't yet dipped my toes into shaders yet, is that the only way of doing it?

[Bigfoot71]

Like this, if that's what you want:

Code: Select all

function love.load()

    canvas = love.graphics.newCanvas()  -- Create a screen-sized canvas

    -- Draw an opaque shape on the canvas

    love.graphics.setCanvas(canvas)
    love.graphics.setColor(1, 1, 1)
    love.graphics.rectangle("fill", 200, 200, 400, 200)
    love.graphics.setCanvas()

    -- Definition of our stencil function which will display a purple rectangle the size of the screen

    stencilFunction = function()
        love.graphics.setColor(1, 0, 1)
        love.graphics.rectangle("fill", 0, 0, love.graphics.getWidth(), love.graphics.getHeight())
    end

end

function love.draw()

      -- Activate the stencil
      love.graphics.stencil(stencilFunction, "replace", 1)
      love.graphics.setStencilTest("greater", 0)

      -- Draw the canvas with a clip defined by the stencil
      love.graphics.draw(canvas, 0, 0)

      -- Turn off the stencil
      love.graphics.setStencilTest()

end

What you need to remember is about these two lines:

Code: Select all

      love.graphics.stencil(stencilFunction, "replace", 1)
      love.graphics.setStencilTest("greater", 0)

"replace" means we want to replace the existing value with the new stencil value (which will be determined by the drawn shape).
And the parameter "1" means the new stencil value that will be assigned to all pixels in the drawing area.

Then for: "love.graphics.setStencilTest("greater", 0)"
"greater" means we want to draw the pixels whose stencil value is above a certain threshold which in this case is "0".

Result we draw what is in the stencil function only in the opaque pixels of the canvas, hoping that this is exactly what you wanted (and hoping to be clear enough, otherwise don't hesitate)

[slime]

How does this work exactly? I haven't yet dipped my toes into shaders yet, is that the only way of doing it?

Yes, you'll need a shader and have the 'discard' keyword inside it when a certain condition is met (such as a pixel's alpha being below a threshold).

This is commonly called alpha testing or alpha clipping, and there's no other way to stop a certain pixel in an object's geometry from affecting stencil or depth buffers while making the rest of the object affect it.

[Bobble68]

Like this, if that's what you want:

Code: Select all

function love.load()

    canvas = love.graphics.newCanvas()  -- Create a screen-sized canvas

    -- Draw an opaque shape on the canvas

    love.graphics.setCanvas(canvas)
    love.graphics.setColor(1, 1, 1)
    love.graphics.rectangle("fill", 200, 200, 400, 200)
    love.graphics.setCanvas()

    -- Definition of our stencil function which will display a purple rectangle the size of the screen

    stencilFunction = function()
        love.graphics.setColor(1, 0, 1)
        love.graphics.rectangle("fill", 0, 0, love.graphics.getWidth(), love.graphics.getHeight())
    end

end

function love.draw()

      -- Activate the stencil
      love.graphics.stencil(stencilFunction, "replace", 1)
      love.graphics.setStencilTest("greater", 0)

      -- Draw the canvas with a clip defined by the stencil
      love.graphics.draw(canvas, 0, 0)

      -- Turn off the stencil
      love.graphics.setStencilTest()

end

What you need to remember is about these two lines:

Code: Select all

      love.graphics.stencil(stencilFunction, "replace", 1)
      love.graphics.setStencilTest("greater", 0)

"replace" means we want to replace the existing value with the new stencil value (which will be determined by the drawn shape).
And the parameter "1" means the new stencil value that will be assigned to all pixels in the drawing area.

Then for: "love.graphics.setStencilTest("greater", 0)"
"greater" means we want to draw the pixels whose stencil value is above a certain threshold which in this case is "0".

Result we draw what is in the stencil function only in the opaque pixels of the canvas, hoping that this is exactly what you wanted (and hoping to be clear enough, otherwise don't hesitate)

Ah this isn't quite what I wanted - maybe it would be helpful if I was clearer with what I want it for.

I'm trying to make some polygons have a pixelated look to them, so what I'm doing is drawing scaled down versions on a canvas, then scaling it back to full size. I then want to use it as a mask for a texture, which will then give me the pixellated polygon with a texture.

Upon writing it out, I've realised there might be a simpler solution that doesn't involve stencils, though it will involve changing a lot of old code (though it would also help fix an issue you might remember I asked about before - I haven't tried to implement any of the fixes yet )

[Bigfoot71]

Ah this isn't quite what I wanted - maybe it would be helpful if I was clearer with what I want it for.

I'm trying to make some polygons have a pixelated look to them, so what I'm doing is drawing scaled down versions on a canvas, then scaling it back to full size. I then want to use it as a mask for a texture, which will then give me the pixellated polygon with a texture.

Upon writing it out, I've realised there might be a simpler solution that doesn't involve stencils, though it will involve changing a lot of old code (though it would also help fix an issue you might remember I asked about before - I haven't tried to implement any of the fixes yet )

I have to admit that you are resourceful but why not rather use meshes and a shader to pixelate the rendering?

Here is an example of a pixelate shader:

Code: Select all

const float pixelSize = 10.0;

vec4 effect(vec4 color, Image tex, vec2 texCoords, vec2 screenCoords) {

    vec2 pixelCoords = vec2(floor(screenCoords.x / pixelSize) * pixelSize,
                            floor(screenCoords.y / pixelSize) * pixelSize);

    return Texel(tex, pixelCoords / love_ScreenSize.xy) * color;

}

[Bobble68]

but why not rather use meshes and a shader to pixelate the rendering?

Mainly because I don't know how to use shaders, and I'm too scared to try learning them

(Don't worry I am fully aware it's probably easier than I think it is)

[Bigfoot71]

Mainly because I don't know how to use shaders, and I'm too scared to try learning them

(Don't worry I am fully aware it's probably easier than I think it is)

I suspected it a little but it's actually much simpler than it seems especially since Löve2d provides a lot of tools to simplify even more the writing of these shaders. (what can be complicated is if you want to make effects that require a lot of mathematical formulas but the language itself is really simple)

If you want to dare to get started (now or another day) there is this blog post which is super useful to start writing shaders for Löve2d and understand how it works: https://blogs.love2d.org/content/beginn ... de-shaders

There is also the wiki page which may help after reading the blog post: https://love2d.org/wiki/love.graphics.newShader

Also having done a bit of C++ or any other typed language can help at the beginning even if it is not essential. (by the way GLSL is much simpler than C++, don't be afraid of what I say ). Otherwise if you are brave you can first start with OpenGL and C++ (if you have already done C++, otherwise you will unfortunately have a hard time), it will require more concentration but once you understand this, learning with Löve2D will go even faster, that's how I did it personally, I don't know if this is the best idea but it worked for me

Others on the forum with even more time of experience may be able to advise you even better, do not hesitate to ask!

But to come back to your problem I think it would be much faster and easier to solve with meshes and shaders (as well as probably more efficient), it's just that it will require a little more math. You already have the shader anyway