LÖVE

Is there a way to enable GL depth testing?

I can move vertices in 3d with the help of a vertex shader but there is no depth testing.

I guess that is what love3d (https://github.com/excessive/love3d) is doing.

Why would you want that in a 2D game?

T-Bone wrote:Why would you want that in a 2D game?

I'd guess so he can ignore draw order and instead use z values.

Alpha blending requires objects to be drawn in back-to-front order for proper blending, so changing the z coordinate using a shader isn't much help in that case.
Also, the hardware depth buffer is binary per-pixel (each pixel of what you draw is either fully occluded or not occluded at all) and uses the geometry of what you draw rather than its opacity (it will treat a fully transparent image the exact same as a fully opaque rectangle), so it's not very useful either, in 2D games.

I'm okay with having 1-bit alpha though which can be achieved with the shader.

I want do do it to try to avoid having to sort thousands of sprites in a Lua-table and to save drawcalls.
Instead I would merge all (static) sprites to one mesh per texture. And let the hardware perform depth test.
In my mind that would yield better performance for a 2D game as well.

Anyway the question was if it can be done.
And as I understand that is part of what the Love3D extension is doing. But maybe I'm wrong?

To reduce the number of draw calls for sprites you should use a [wiki]SpriteBatch[/wiki], it's the most efficient way to do that in LÖVE currently.

The love3d library will let you do depth testing and depth writes. However, if you output the depth value in a pixel shader via checking the alpha value in a texture, you'll probably get significantly worse performance than if you had no depth at all.

Having a conditional operation per-pixel based on the texture's alpha will slow down the pixel shader a lot, and by writing the depth value in the pixel shader you'll be preventing the GPU from doing an important optimization (normally it skips processing of any pixel that it knows is occluded based on depth, but if you write the depth in the pixel shader it can't know it's occluded beforehand.)

I doubt it will run any faster using depth the depth buffer compared to sorting CPU-side, if you're using a sprite batch. You could probably find plenty of ways to optimize the CPU-side sorting too, if that becomes a bottleneck.