Making an outline shader for text

[progress]

Hi all,

I have virtually no knowledge with GLSL so I've been using ChatGPT to try to make an outline shader for my Love2D project, but it doesn't seem to be working, no matter how much troubleshooting I do.

If you're familiar with GLSL, I'd really appreciate some corrections to this shader code, or perhaps the way I'm using it!

Here's the shader code

Code: Select all

outlineShader = love.graphics.newShader([[
    vec4 position(mat4 transform_projection, vec4 vertex_position)
    {
        return transform_projection * vertex_position;
    }
]], [[
    extern vec2 outlineSize;   // the size of the outline (in pixels)
    extern vec4 textColor;    // the color of the text
    extern vec4 outlineColor; // the color of the outline
    
    vec4 effect(vec4 color, Image texture, vec2 texture_coords, vec2 screen_coords)
    {
        // calculate the outline color
        vec4 outline = vec4(0.0);
        for (float x = -outlineSize.x; x <= outlineSize.x; x += 1.0) {
            for (float y = -outlineSize.y; y <= outlineSize.y; y += 1.0) {
                float factor = (abs(x) + abs(y)) / (outlineSize.x + outlineSize.y);
                outline += Texel(texture, texture_coords + vec2(x, y) / love_ScreenSize.xy) * factor;
            }
        }
        outline = mix(outlineColor, outline, outline.a); // mix the outline color with the calculated color
    
        // calculate the text color
        vec4 text = Texel(texture, texture_coords);
    
        // blend the outline and text color
        vec4 result = mix(outline, text, step(0.0, text.a)); // use the alpha channel of the text to create a mask
        result = mix(textColor, result, step(0.0, result.a)); // mix the result with the text color
    
        return result * color;
    }
]])

It has gone through many iterations too, and changed quite dramatically. Please let me know if you'd like to see another version of the code, this is the first and only version of ChatGPT's code that produces no error. After defining

Code: Select all

outlineShader

I do this:

Code: Select all

outlineShader:send('outlineSize', {2, 2}) -- the size of the outline in pixels
outlineShader:send('textColor', {0, 0, 0, 1}) -- the color of the text (white)
outlineShader:send('outlineColor', {0, 1, 0, 1}) 

Which to my understanding, is pretty self explanatory in what it does. Then in my love.draw() function, I do this (for testing purposes):

Code: Select all

love.graphics.setShader(outlineShader)
love.graphics.print('Hello, World!)
love.graphics.setShader()

And it just prints the text as usual, white with white outlines. However if I made the outlineSize incredibly high, it greatly hinders the performance of the game, so I presume the shader is doing something.

Any and all help is greatly appreciated!

Also, let me know if you need more context in order to help me out!

[Bigfoot71]

Having chatGPT do something entirely is going to be a waste of your time, it may help you save time on things you already know, but learning with chatGPT or having it do something entirely that you don't know how to do is going to be source of a lot of problems for you unfortunately.

It is a good tool certainly but to use in good conditions.

Here is how I would have made such a shader personally:

Code: Select all

local lg = love.graphics

local shader = lg.newShader[[

    extern vec4 outline_color;
    extern float outline_width;

    vec4 effect(vec4 color, Image tex, vec2 tex_coords, vec2 screen_coords) {

        vec4 pixel = Texel(tex, tex_coords);

        if (pixel.a == 0.) discard;

        float dist = pixel.a;
        for (float i = 1.; i <= outline_width; i++) {
            dist = min(dist, Texel(tex, tex_coords + vec2(i, 0.)).a);
            dist = min(dist, Texel(tex, tex_coords + vec2(-i, 0.)).a);
            dist = min(dist, Texel(tex, tex_coords + vec2(0., i)).a);
            dist = min(dist, Texel(tex, tex_coords + vec2(0., -i)).a);
        }

        if (dist > 1.0 - outline_width) {
            return mix(outline_color, color, pixel.a);
        }

        return pixel*color;

    }

]]

local font = lg.newFont(64)

function love.draw()

    shader:send("outline_color", {1,1,1,1})
    shader:send("outline_width", 2)

    lg.setShader(shader)

        lg.setFont(font)
        lg.setColor(0,0,0)  -- Text color
        lg.print("Hello World !",200,200)

    lg.setShader()

end

In this shader you set the text color with love.graphics.setColor() and the outline color and size with shader:send()
Here is the result, you could have an even cleaner if the text is rendered in a canvas, because the shader is only applied within the limit of the "image" of the letters, and also by modifying the shader a bit:


You can also do it without a shader by doing something like this for example:

Code: Select all

local lg = love.graphics

local function genOutlined(outlineColor, textColor, text, font, ol)

    font = font or lg.getFont()
    local textObj = lg.newText(font, text)

    local textImg = lg.newCanvas(
        textObj:getWidth()+ol*2,
        textObj:getHeight()+ol*2
    )

    lg.setCanvas(textImg)

        lg.setColor(outlineColor)

        local n = -ol

        for i=1,2 do
            lg.draw(textObj, ol, ol+n)
            lg.draw(textObj, ol+n, ol)
            lg.draw(textObj, ol-n, ol+n)
            lg.draw(textObj, ol+n, ol-n)
            n = -n
        end

        lg.setColor(textColor)
        lg.draw(textObj, ol, ol)

    lg.setCanvas()

    textObj:release()

    return textImg

end

local font = lg.newFont(64)
local textCanvas = genOutlined({1,1,1},{0,0,0},"Hello World !",font,2)

--local textImg = lg.newImage(textCanvas:newImageData())
-- textCanvas:release()

function love.draw()
    lg.setColor(1,1,1)
    lg.draw(textCanvas,200,200)
end

Here's what it looks like, you can see a slight defect in the outer corners of the letters, it's probably fixable but I'll let you do the rest of the work (the shader that I also shared by the way)


Or yet another possible version of this function that displays the text directly, however this will mishandle the use of the alpha channel, just like the above example in a different way:

Code: Select all

local function outlined(outlineColor, textColor, text, font, x, y, ol, r, sx, sy, ox, oy, kx, ky)

    font = font or lg.getFont()
    local textObj = lg.newText(font, text) -- Can be optimized if you define it once then you redefine it with :set(text)

    lg.setColor(outlineColor)

    local n = -ol
    for i=1,2 do
        lg.draw(textObj, x, y+n, r, sx, sy, ox, oy, kx, ky)
        lg.draw(textObj, x+n, y, r, sx, sy, ox, oy, kx, ky)
        lg.draw(textObj, x-n, y+n, r, sx, sy, ox, oy, kx, ky)
        lg.draw(textObj, x+n, y-n, r, sx, sy, ox, oy, kx, ky)
        n = -n
    end

    lg.setColor(textColor)
    lg.draw(textObj, x, y, r, sx, sy, ox, oy, kx, ky)

    textObj:release()

end

It's up to you now to use the method you want and of course adapted to your sauce because what I'm sharing with you remains as an example. If you need help understanding what I have shared with you, don't hesitate!

[pgimeno]

Instead of ChatGPT, have you tried the forum's search function?

You could have run into this: https://love2d.org/forums/viewtopic.php ... 15#p221215

[RNavega]

With Löve's text meshes specifically --which are directly used with love.graphics.newText(), or indirectly used with .print() or .printf()-- it's difficult to add an outline effect through a shader, because the glyphs come from a very efficiently-packed atlas texture with little padding (edit: reference source in here).

So you'll run into this problem: if you add a thick enough outline, it might get clipped off by the glyph quad, especially in narrow glyphs like the capital "i".
If you try to remedy that by expanding the glyph quad vertices in a vertex shader, you'll end up rendering parts of other glyphs in the expanded quad because as mentioned, they're tightly packed and the quad will start sampling them.

The alternatives to get styled text with an outline are:
A) Implementing your own text rendering, like using an (M)SDF font.
B) Using an image font with the outlines already rasterized onto the glyphs.

Option B is very easy to do and works great, like using BMFont to generate the image font. Read more in the wiki: https://love2d.org/wiki/love.graphics.newImageFont

[RNavega]

if you add a thick enough outline, it might get clipped off by the glyph quad, especially in narrow glyphs like the capital "i".
If you try to remedy that by expanding the glyph quad vertices in a vertex shader, you'll end up rendering parts of other glyphs in the expanded quad because as mentioned, they're tightly packed and the quad will start sampling them.

Quoting myself here, I wonder if there isn't a way to prevent that, like implementing some manual texture clamping so you can expand the glyph quad in the vertex shader, and still have it not sample other neighboring glyphs in the texture. This will only be possible if you know the original size of the quad, so you can do a "if the UV of the expanded quad falls outside of the original quad, then ignore the sampled texture".

[slime]

The alternatives to get styled text with an outline are:
A) Implementing your own text rendering, like using an (M)SDF font.
B) Using an image font with the outlines already rasterized onto the glyphs.

If you use a BMFont, love respects any padding specified there and you can use (M)SDF font textures with love's Font objects. SDF rendering lets you generate an outline really simply in a shader (so you don't need to pre-generate the outline if you don't want), and there are plenty of tools to convert a ttf file to a BMFont.

[RNavega]

If you use a BMFont, love respects any padding specified there and you can use (M)SDF font textures with love's Font objects.

Ah, that's very cool. You're saying, "use Löve's BMFont support to draw from a SDF or MSDF font texture".

So the steps would be:
- Get that distance texture in BMFont format (a pair with metadata file and texture atlas file), so it can be loaded with love.graphics.newFont(metadataFile, atlasFile).
- When drawing some text, turn on your SDF or MSDF shader with love.graphics.setShader, then draw the text with any of print/printf/draw(textObject).

For tools, I know that you can use MSDF-Atlas-Gen to generate the atlases from a TTF/OTF font file. Besides the atlas image, it outputs the metadata as JSON, CSV or other formats, but not the BMFont FNT metadata.
Still, I think I can take the JSON that it outputs and write a tool that ports it to BMFont's FNT, using these references:
- The values used in the MSDF JSON file
- The FNT BMFont format
- Additional description of the FNT tags

As you said, with an SDF font you can do effects at the shader level: bold, pseudo-italic (by shearing the glyph quads horizontally in the vertex shader), outline, glow, drop-shadow. It's definitely worth looking into.

[Bigfoot71]

Since what I proposed is too "theoretical" and would not be good to apply in a real project, I still tried to make functions that did not use a shader to avoid needing using another font format (even though it's probably the best solution for now), it also fixes the problem of missing outer corners.

First of all the most basic to understand the principle:

Code: Select all

return function (text, x, y, ol, c1, c2, r, sx, sy, ox, oy, kx, ky)

    local lg = love.graphics
    local p = lg.print

    lg.setColor(c1)
    for dx = -ol, ol do
        for dy = -ol, ol do
            if dx ~= 0 or dy ~= 0 then
                p(text, x + dx, y + dy, r, sx, sy, ox, oy, kx, ky)
            end
        end
    end

    lg.setColor(c2)
    p(text, x, y, r, sx, sy, ox, oy, kx, ky)

end

And another more worked, with the choice to put the colors as a table or individual numeric values:

Code: Select all

local lg = love.graphics
local d = lg.draw

local textObj = lg.newText(
    lg.getFont()
)

return function (text, x, y, ol, ...)

    local r, sx, sy, ox, oy, kx, ky;
    local c1, c2, r1, g1, b1, a1, r2, g2, b2, a2;

    if type(select(1, ...)) == "table" then
        c1, c2, r, sx, sy, ox, oy, kx, ky = ...
    else
        r1, g1, b1, a1, r2, g2, b2, a2, r, sx, sy, ox, oy, kx, ky = ...
    end

    textObj:set(text)
    textObj:setFont(lg.getFont())

    lg.setColor(
        r1 or c1[1],
        g1 or c1[2],
        b1 or c1[3],
        a1 or c1[4]
    )

    local minX = x - ol
    local minY = y - ol
    local maxX = x + ol
    local maxY = y + ol

    for dx = minX, maxX do
        if dx == minX or dx == maxX then
            for dy = minY, maxY do
                d(textObj, dx, dy, r, sx, sy, ox, oy, kx, ky)
            end
        else
            d(textObj, dx, minY, r, sx, sy, ox, oy, kx, ky)
            d(textObj, dx, maxY, r, sx, sy, ox, oy, kx, ky)
        end
    end

    lg.setColor(
        r2 or c2[1],
        g2 or c2[2],
        b2 or c2[3],
        a2 or c2[4]
    )

    d(textObj, x, y, r, sx, sy, ox, oy, kx, ky)

end

I did not find a solution optimized enough for alpha channel management, I tried different BlendModes in doubt but I never had the expected result.

I also made performance tests that I put in attachment, the simplified version on 10000 iterations gives on my PC an average of 0.000130 seconds and that more worked gives 0.000081 seconds.

Suggestion: It could be nice to add the means to make texts with outline with lg.printf or other, after all almost all the other engines allow it