problem using stencil function

[DebasishDatta]

Code: Select all

function love.draw()

	push:start()
		love.graphics.setCanvas({canvas,stencil=true})
		love.graphics.stencil(function()
			love.graphics.rectangle('fill',VIRTUAL_WIDTH/2 - 40,VIRTUAL_HEIGHT/2 - 10, 30,30)
		end,'replace',1,false)
		love.graphics.setStencilTest('less',1)
		
		love.graphics.draw(Ctexture,Cquads[1],posX,posY)
		
		love.graphics.setStencilTest()
		love.graphics.setCanvas()

	push:finish()

end

Here I am trying to use stencil and my target is, when the character is inside the stencil area he should not be visible and for rest area normal screen display will take place, i.e character will be displayed in the screen normally, but the result of above code is something different, it is drawing the character in the stencil area. As a result small display is taking place with the character that to the area drawn by the rectangle inside stencil function. kindly help.

[pgimeno]

You can use other compare mode instead of 'less', check CompareMode. If you want the opposite of what you currently have, you can try 'gequal'.

[DebasishDatta]

Thank you for your reply, I have tried with 'gequal', but it does not work, I thing the problem is that the stencil buffer is not getting synchronized with canvas, is there any other way to set stencil = true other than using setCanvas? or other way to use the stencil?

[pgimeno]

Can you post a complete runnable example? It doesn't need to be your full project, just enough code to be able to run it and see the problem. It might be easy from the code above, but I don't have time right now to try, and after all you're the one who wants help

[DebasishDatta]

Code: Select all

push = require 'push'
VIRTUAL_WIDTH  = 384
VIRTUAL_HEIGHT = 216
WINDOW_HEIGHT =  720
WINDOW_WIDTH  = 1280
WALK_SPEED = 40
function love.load()
	push:setupScreen(VIRTUAL_WIDTH,VIRTUAL_HEIGHT,WINDOW_WIDTH,WINDOW_HEIGHT,{
		fullscreen = false,
		vsync = true,
		resizable = true
	})
	Ctexture = love.graphics.newImage('entities.png')
	Cquads = GenerateQuads(Ctexture,16,16)
	posX = VIRTUAL_WIDTH/2
	posY = VIRTUAL_HEIGHT/2
	love.keyboard.keyPressed = {}
	canvas = love.graphics.newCanvas()
end

function love.keypressed(key)
	love.keyboard.keyPressed[key] = true
end

function love.keyboard.wasPressed(key)
 return love.keyboard.keyPressed[key]
end

function love.update(dt)
   if love.keyboard.wasPressed('escape') then
	love.event.quit()
   end
   if love.keyboard.isDown('up') then
	posY = posY - WALK_SPEED * dt
   end
   if love.keyboard.isDown('down') then
	posY = posY + WALK_SPEED * dt
   end
   if love.keyboard.isDown('left') then
	posX = posX - WALK_SPEED * dt
   end
   if love.keyboard.isDown('right') then
	posX = posX + WALK_SPEED * dt
   end
   love.keyboard.keyPressed = {}
end

function love.draw()

	 push:start()
		
		 love.graphics.setCanvas{canvas,stencil = true}		
	     	 --love.graphics.setCanvas(canvas)
		love.graphics.stencil(function()
			love.graphics.rectangle('fill',VIRTUAL_WIDTH/2 - 40,VIRTUAL_HEIGHT/2 - 10, 60,60)
		end,'replace',1,false)
		love.graphics.setStencilTest('gequal',0)
		love.graphics.draw(Ctexture,Cquads[1],posX,posY)
		love.graphics.setStencilTest()
		love.graphics.setCanvas()
		--love.graphics.draw(Ctexture,Cquads[1],posX,posY)	
		
		

	push:finish()

end

function GenerateQuads(atlas,tileWidth,tileHeight)
	local sheetWidth  = atlas:getWidth()/tileWidth
	local sheetHeight = atlas:getHeight()/tileHeight
	local sheetCounter = 1
	local spriteSheet  = {}
	for y = 0, sheetHeight - 1 do
		for x = 0, sheetWidth - 1 do
			spriteSheet[sheetCounter] = love.graphics.newQuad(x * tileWidth, y * tileHeight,
				tileWidth, tileHeight, atlas:getDimensions())
			sheetCounter = sheetCounter + 1
		end
	end
	return spriteSheet
end

In the above code I have tried to do exactly what I want, I want that when character enters the area defined by the rectangular area inside stencil function does not render to the screen. I have other alternative, in program I can set a flag for this, but I want to use stencil for this. Can you please help.

[ReFreezed]

Your original code with love.graphics.setStencilTest('less',1) is correct if you want to draw everywhere except in that rectangle. However, you draw Ctexture to a canvas but you don't draw the canvas anywhere after that (which makes your wording confusing because you shouldn't be able to see anything at all ever).

Also note that push:start() sets its own canvas, and when you call love.graphics.setCanvas() you potentially ruin whatever the push library is doing. You can solve this by calling love.graphics.push("all") before you call love.graphics.setCanvas(canvas), and call love.graphics.pop() instead of love.graphics.setCanvas(). However, it's not clear from the given code if you need your own canvas here at all as, by looking at the push library code, stencils are enabled by default when calling push:start().

(Also, next time you post your code, please post a runnable example - e.g. a .love file that includes libraries and images being used. It just wastes everyone's time trying to guess what is needed to get the code to run.)

Edit: Added correction to error pointed out by pgimeno.

[pgimeno]

You can solve this by calling love.graphics.push()

Correction: love.graphics.push("all")

before you call love.graphics.setCanvas(canvas), and call love.graphics.pop() instead of love.graphics.setCanvas().

But I recommend saving the canvas and restoring it instead of using push/pop:

Code: Select all

local saveCanvas = love.graphics.getCanvas()
love.graphics.setCanvas(myCanvas)
-- draw stuff, then:
love.graphics.setCanvas(saveCanvas)

I haven't looked into the code to be sure, but love.graphics.push("all") is probably somewhat overkill when only a handful of things have changed.

[DebasishDatta]

Thank you pgimeno and ReFreezed , your suggestions were very helpful, restoring the canvas solved my problem and my program has started behaving the way I wanted. I understand that, I have wasted your time by not posing .love file along with the .png files. In future post I will certainly do that. I have a little doubt here, By using getCanvas() whether we are getting the reference of the canvas set by push? and after I draw the graphics again I am restoring the canvas initiated by push function? Is my understanding correct? if not can you please explain which canvas we are getting by using getCanvas() function. In this context I want to mention that, I have not used push:start("all")

[ReFreezed]

If we look at the definition of push:start() we see that it calls love.graphics.setCanvas(), so when you call love.graphics.getCanvas() after calling push:start() you get the canvas that push:start() set. So yes, you understand correctly.

push:start("all") is not a thing - the push:start() function does not take any arguments. I'm assuming you meant love.graphics.push("all").