LÖVE

Hello everyone,

I recently started working on a 3d game prototype featuring portals, which would ideally work like in the game of the same name. I eventually managed to get the rendering of portals to work more or less properly (even recursive portals), but I have the following two issues:

1. I initially wanted to apply the algorithm described here, but at the very end (before the "Code" section), it requires testing the geometry that will be drawn to the stencil against the current stencil values. It seems two people have asked about this before (link 1, link 2), but with no conclusive answers. I have done tests, and it seems that calling love.graphics.setStencilTest within the callback passed to love.graphics.stencil does not work, and looking at the LÖVE source code (specifically Graphics::setStencilTest in this file) seems to confirm this.
   My question is: is there a sneaky way to do this, or is it impossible with the current LÖVE API, and should I file a bug report/feature request or not?
   (FYI, I went around this issue by allocating a different stencil buffer for each possible visible portal instance, but that's increeeedibly inefficient.)
2. This is unrelated to LÖVE specifically, but since I'm posting my code here, I might as well ask; in my current implementation, the stencil pattern marking portals within portals seems to partially disappear under certain viewing angles, and I can't for the life of me figure out what is erasing it. If anyone wants to take a stab at it, they are very welcome to.
   
   Screenshot of the problem:
   
   
   The .love is attached. To move, use WASD and the mouse. You can grab and move the cubes, as well as the portal in front of the starting position.
   The most relevant file is 3d.lua, especially the renderView function, as well as shaders.lua for the GLSL code. I apologize in advance for the uncommentated code.

Luapix wrote: ↑Thu Jul 04, 2019 7:45 pmin my current implementation, the stencil pattern marking portals within portals seems to partially disappear under certain viewing angles, and I can't for the life of me figure out what is erasing it.

It doesn't look to me like something is erasing it. It looks more like for some reason, some triangles are not being drawn (culled?).

Disclaimer: I have not actually looked at the code.

Awesome effect, by the way. Kudos.

(Sorry for the very, very late response, pgimeno, I thought I'd get an email when somebody responds to the thread...)

Basically, when I render a scene view (either the main camera's view, or the view through a particular portal), I first render the scene normally, where the portal's surface is just a sky-colored rectangle, while making sure to also render said surface to the stencil. I then render the next portal views to different canvases recursively, and draw the result of that in the stenciled part.

The weird thing is that the portal's surface and the inner portal view are getting rendered correctly separately: the portal's surface being drawn is the reason why the disappearing parts are sky-colored and we don't just see through the portal frame, and the part which vanishes is being rendered correctly in the inner view, so I really don't think it is a culling problem.

The problem seems to be that the stenciled part of the canvas doesn't correspond to the portal's surface (even though that's literally the geometry it's drawn from), so only part of the inner view gets rendered onto the surface. The only possible conclusion is that the stencil is getting erased some time after it's been drawn. Moreover, the only time when the stencil should get erased is when an object is rendered in front of the portal's surface. But as you can tell in the screenshot, nothing is in front of the portal's surface, and I have no idea what kind of weird invisible moving object would have to be there to get the moving vanishing patterns I observe.

I've stopped working on this project and have moved on to other things, so it's not a big deal. It would be cool to figure out what the issue was though. I've been working on some OpenGl projects in C++ recently, so I might try to recreate my Lua code one for one to see if the issue is still there. (I'd also to be able to resolve problem #1 as well.)

Luapix wrote: ↑Sat Aug 03, 2019 12:32 pm (Sorry for the very, very late response, pgimeno, I thought I'd get an email when somebody responds to the thread...)

By default you don't get emails if someone quotes you, but you do if you subscribe to the thread. I think that can be changed in your preferences.

Luapix wrote: ↑Sat Aug 03, 2019 12:32 pmThe weird thing is that the portal's surface and the inner portal view are getting rendered correctly separately: the portal's surface being drawn is the reason why the disappearing parts are sky-colored and we don't just see through the portal frame, and the part which vanishes is being rendered correctly in the inner view, so I really don't think it is a culling problem.

That was a shot in the dark, really. I hadn't looked into the article you linked or the code. Now that I have, I've noticed this sentence at the very end:

• Here’s the resulting code on GitHub. There are still various edge-cases that cause weird behavior, so feel free to open a pull request if you manage to improve something

So I downloaded his demo and tried it, and this is what I got:
The black background and the red lines are not in the scene. So, his C++ demo has the same problem as your Love2D demo. Therefore something fails with both. What can it be?

Well, from his description:

• To obscure the part of the portal on level two, we have to enable the stencil test for the part where we render our portal to the stencil test.

I don't think this can work in parallel (which is what GPUs do). Many people, including me, have had similar problems when trying to write to a canvas while they're drawing the same canvas. Here's my case: https://love2d.org/forums/viewtopic.php?f=4&t=81251

The solution was to use two canvases as a kind of double buffer, drawing one of them into the other and then copying and starting again. I bet the same needs to be done here for the stencil buffers to work as designed.

Luapix wrote: ↑Sat Aug 03, 2019 12:32 pm The only possible conclusion is that the stencil is getting erased some time after it's been drawn.

My guess is that it's not yet drawn rather than erased, due to parallelism issues. It has some resemblance to the famous i++ + ++i; in C, which causes undefined behaviour because there isn't a sequence point between the first modification of i and the second. In your case, there isn't a "sequence point" between the reading and the writing of the stencil buffer, and I believe that OpenGL is "authorized" to do these in parallel, which means the stencil may not be completely drawn while it's being used.

Now there's a problem, related to #1. If you use the stencil drawing function, the problem is that it is called with the stencil already active. In the third recursive call, you should be using the first stencil, which is already active and therefore it may cause issues.

There are two possible ways around that. One is to have one stencil buffer per nesting level; the other is to ignore the stencil drawing function and activate/deactivate the stencil manually instead. I think the latter is possible by using a canvas like a stencil, but the docs on that were still poor the last time I checked. Maybe that has changed now.

pgimeno wrote: ↑Sat Aug 03, 2019 2:53 pm There are two possible ways around that. One is to have one stencil buffer per nesting level; the other is to ignore the stencil drawing function and activate/deactivate the stencil manually instead.

In the end, I went for the first option; each nesting level has a stencil buffer + canvas. This also means I don't need to "enable the stencil test for the part where we render our portal to the stencil test", which was problem #1: I just draw the inner scene to another canvas, then draw *that* to the previous canvas using the previous stencil.

pgimeno wrote: ↑Sat Aug 03, 2019 2:53 pm

Luapix wrote: ↑Sat Aug 03, 2019 12:32 pm The only possible conclusion is that the stencil is getting erased some time after it's been drawn.

My guess is that it's not yet drawn rather than erased, due to parallelism issues. It has some resemblance to the famous i++ + ++i; in C, which causes undefined behaviour because there isn't a sequence point between the first modification of i and the second. In your case, there isn't a "sequence point" between the reading and the writing of the stencil buffer, and I believe that OpenGL is "authorized" to do these in parallel, which means the stencil may not be completely drawn while it's being used.

I doubt that OpenGL and LÖVE are so badly designed that you would be able to use the stencil before it has finished being drawn; in that case, LÖVE would require some kind of syncing primitive to even be able to draw something within a stencil. And even if that was the case, considering that I draw the inner scenes between setting the stencil and actually using it, I think enough time has elapsed that it should be done. The tests that I've mentionned involved visualizing the stencil values for each canvas/stencil of each nesting level, instead of rendering the scene; by that time, the stencil values would definitely be drawn.

pgimeno wrote: ↑Sat Aug 03, 2019 2:53 pm The black background and the red lines are not in the scene. So, his C++ demo has the same problem as your Love2D demo.

Finally, I doubt those are the same problem. In the demo, the buggy part shows an object rendered in perspective *from the other side of the portal*; ie. it's part of the scene within the portal. My problem is that part of the scene within the portal isn't being rendered onto the portal surface at all, and we instead see the portal surface's actual color, before any inner scene is rendered above it. In fact, if I make the portal's surface transparent (which I didn't do for the screenshot because it makes things confusing), the "erased" part of the portal lets us see *through* the portal frame to the other side of the scene.

I will try to take a look at the demo nonetheless, see if I can see anything similar.

Luapix wrote: ↑Sat Aug 03, 2019 4:00 pm I doubt that OpenGL and LÖVE are so badly designed that you would be able to use the stencil before it has finished being drawn;

Well, it definitely happens happened with canvases; see the thread I linked (and I was not the first nor the last one having the issue). I don't see why it can't happen with stencils.

Luapix wrote: ↑Sat Aug 03, 2019 4:00 pmIn that case, LÖVE would require some kind of syncing primitive to even be able to draw something within a stencil.

Feel free to ask for something like that in the issue tracker, but I don't know how feasible will that be, and if it is, how willing would the authors be to implement it.

Edit: Canvases now emit a warning: "Error: canvas.lua:25: Cannot render a Canvas to itself!". Maybe it's not too hard.

Luapix wrote: ↑Sat Aug 03, 2019 4:00 pmAnd even if that was the case, considering that I draw the inner scenes between setting the stencil and actually using it, I think enough time has elapsed that it should be done.

I wouldn't count on that.

pgimeno wrote: ↑Sat Aug 03, 2019 2:53 pm Finally, I doubt those are the same problem. In the demo, the buggy part shows an object rendered in perspective *from the other side of the portal*; ie. it's part of the scene within the portal.

Maybe. The fact that the demo runs at ~5 FPS in my machine doesn't help.