Problem with hanging and reading stdout from python in threads

[yoki]

I just posted about a video corruption problem I'm having on the RPi, but I also have a problem reading the output of a python script.

I have some sensors attached to the Raspberry Pi which I'm interfacing with several python scripts. When run on their own they output normally to stdout. When writing to a pipe I already found out I need to flush the buffer or it won't be seen by a plain lua script.

Code: Select all

python3 -u script.py

print(str, flush=True)

sys.stdout.write(str)
sys.stdout.flush()

Any of those three will do. However when I open the python program from a love.thread the behavior is very erratic: sometimes it opens normally and passes along the lines from the pipe, and sometimes the thread hangs indefinately without error.

This is the minimal thread code

Code: Select all

print("motion_thread: starting...")

local p = assert(io.popen("python3 /home/pi/zapview/python/motion_detection.py", "r"))
if not p then
	print("motion_thread: motion_detection.py error")
end
print("motion_thread: motion_detection.py process: " .. tostring(p))

while p do
	local line = p:read("*l")
	if line then
            print("motion_thread: "..tostring(line))
	    love.thread.getChannel("motion"):push(true)
	end
end

p:close()

This either works or it hangs on the io.popen call. I am also using mode2 from the lirc package to capture the raw output of a remote, with this same thread code, and this works very well, no hanging at all.

It mostly seems to appear when I'm using more then one of those threads (different files launching different scripts). And when they hang they hang the whole love game unless I put in a sleep at the start of each thread

Code: Select all

local function sleep(n)
	os.execute("sleep " .. tonumber(n))
end
sleep(5)

I also tried using an intermediate file to write to

Code: Select all

local f = os.tmpname()
printf("motion_thread: using tmp file: "..f)
os.execute("python3 /home/pi/zapview/python/motion_detection.py > "..f)
for line in io.lines(f) do
	printf("motion_thread: "..tostring(line))
	love.thread.getChannel("motion"):push(true)
end
os.remove(f)

This does launch the process every time and it does write to the tmp file correctly (I can tail -f it and the output is piping in) however the tread is not reading any lines from the file!

The most minimal python script I can reproduce this with

Code: Select all

#!/usr/bin/env python3

import sys
import time

while True:
    # print("test")
    sys.stdout.write('test')
    sys.stdout.flush()
    time.sleep(1)

Some other thing I noticed, after using print in a thread an io.flush() is needed to make it show up in the love output. Not sure if this is relevant, it doesn't seem to affect the hanging either way.

I've compiled love from the love-xxx-linux-src.tar.gz files in the bitbucket repo with configure and make. I'm getting the same behavoir from 11.0, 11,1 and 11.2.

I'm stuck with this problem for more then a week now, and it's a game for a video art installation that is going to be displayed in a few days, so if anyone is able to help me solve this or work around it that would be greatly appreciated.

[keharriso]

Code: Select all

#!/usr/bin/env python3
import sys
import time

while True:
    # print("test")
    sys.stdout.write('test')
    sys.stdout.flush()
    time.sleep(1)

First check: are you actually printing out complete lines in your test cases? The above code doesn't print out any "\n" characters, which means p:read("*l") will naturally freeze forever, waiting for an end-line that never comes.

Another thing: it'd be really helpful if you could provide a full archive of the minimum code (the script.py, the main.lua, and the thread.lua) that you want to get working.

I tried your examples with this main.lua and it seemed to be working fine:

Code: Select all

function love.load()
	local thread = love.thread.newThread("thread.lua")
	thread:start()
end

local x = 0

function love.update(dt)
	local msg = love.thread.getChannel("thread"):pop()
	if msg then
		print("read "..x)
		x = x + 1
	end
end

[yoki]

Thanks for the answer!

First check: are you actually printing out complete lines in your test cases? The above code doesn't print out any "\n" characters, which means p:read("*l") will naturally freeze forever, waiting for an end-line that never comes.

Oh yes sorry, that's a copy-paste error. I tested the real scripts with a + '\n' at the end and now use print(str, flush=True). Note however that the thread doesn't seem to hang on p:read() because it never prints the status messages after assert(io.popen()).

Another thing: it'd be really helpful if you could provide a full archive of the minimum code (the script.py, the main.lua, and the thread.lua) that you want to get working.

Ok I've managed to create the minimal love game that shows what I mean. There are 6 threads started at startup, 3 of which contain a sleep(20) in the python code before the main loop to simulate the init time of the sensors. One would expect the 6 python processes to launch all at once (check with "ps a" on unix) and see output of 3 threads and the delayed threads output after 20 seconds.

However this is not what happens, the behavior is a bit random, but usually the python processes (and the output in the main window and console) only start appearing after 20 seconds and then slowly trickle in until they are all running and producing output. This can take as long as 40-60 seconds.

I found this out when after bothering all the sensors to produce output the threads all started to produce output after a few minutes. So I have a workaround, but it still puzzles me why this happens.

[keharriso]

OK, I found success by using Lua coroutines instead of LÖVE threads. Check out the attached code. You'll need to install the "luaposix" library (easy with LuaRocks). Bonus: there's no child threads or processes, it's all contained in a single thread.

[yoki]

Awesome, the example seems to work fine, I'll check it out with the real code tomorrow.

Btw, do you by any chance know how I could detect video corruption and reload it live(ish)? As that's the other problem I'm currently struggling with: viewtopic.php?f=4&t=86386.

[yoki]

Not sure if should mark this thread as solved, as I still don't know the cause and it could be a bug in love2d.

[pgimeno]

I was studying it for a while, but I couldn't come to any conclusion. It seems to me like a locking/mutex problem of some sort. There are too many factors involved.

[keharriso]

I posted an issue on BitBucket here. Given the obscurity of the bug, I'd be surprised to get a solution any time soon, but I guess you never know.

[slime]

As per the bitbucket issue thread, I believe Lua's implementation of popen won't always run concurrently if there is a long-running popen call currently active. I don't think it has anything to do with love's own code, aside from the fact that love exposes threads for you to use.

[pgimeno]

It seems to be related to LuaJIT, but no idea how. Recompiling LÖVE with PUC Lua made the example work fine. Switching to LuaJIT made it fail. Commenting out fflush(NULL) from the LuaJIT popen implementation did not help.