Is multi-thread a ram hog, or is this a memory leak?

[toralord]

I'm trying to lrean about mutli-threading and I think I found a bug working with tables.
It seems to to have a memory leak when iterating though table and setting is values.
When I take out the table I get no memory leak, and my ram don't shoot up using 32 GB the
"the max of my computer " forcing a memory purge.


This is my main code.

Code: Select all

function love.load()
    coco = {}
    for i = 1, 100000 do
        coco[i] = 0
    end

    thread = love.thread.newThread("thread.lua")
    channel = love.thread.getChannel("test")
    thread:start(coco)
    i = {}
    i[1]='thread 1 is work'
    
    thread2 = love.thread.newThread("thread2.lua")
    channel2 = love.thread.newChannel("test2")
    thread2:start(coco)
    i2 = {}
    i2[1]='thread 2 is work'
    
    thread3 = love.thread.newThread("thread3.lua")
    channel3 = love.thread.newChannel("test3")
    thread3:start(coco)
    i3 = {}
    i3[1]='thread 3 is work'
    
    thread4 = love.thread.newThread("thread4.lua")
    channel4 = love.thread.newChannel("test4")
    thread4:start(coco)
    i4 = {}
    i4[1]='thread 4 is work'

    thread5 = love.thread.newThread("thread5.lua")
    channel5 = love.thread.getChannel("test5")
    thread5:start(coco)
    i5 = {}
    i5[1]='thread 5 is work'

    thread6 = love.thread.newThread("thread6.lua")
    channel6 = love.thread.getChannel("test6")
    thread6:start(coco)
    i6 = {}
    i6[1]='thread 6 is work'

    thread7 = love.thread.newThread("thread7.lua")
    channel7 = love.thread.getChannel("test7")
    thread7:start(coco)
    i7 = {}
    i7[1]='thread 7 is work'

    thread8 = love.thread.newThread("thread8.lua")
    channel8 = love.thread.getChannel("test8")
    thread8:start(coco)
    i8 = {}
    i8[1]='thread 8 is work'

    thread9 = love.thread.newThread("thread9.lua")
    channel9 = love.thread.getChannel("test9")
    thread9:start(coco)
    i9 = {}
    i9[1]='thread 9 is work'
end
function love.update(dt)
    v = channel:pop()
    if v then
        i[1]=#v
    end
    
    v2 = channel2:pop()
    if v2 then
        i2[1]=v2
    end
    
    v3 = channel3:pop()
    if v3 then
        i3[1]=v3
    end
    
    v4 = channel4:pop()
    if v4 then
        i4[1]=v4
    end

    v5 = channel5:pop()
    if v5 then
        i5[1]=#v5
    end

    v6 = channel6:pop()
    if v6 then
        i6[1]=#v6
    end

    v7 = channel7:pop()
    if v7 then
        i7[1]=#v7
    end

    v8 = channel8:pop()
    if v8 then
        i8[1]=#v8
    end

    v9 = channel9:pop()
    if v9 then
        i9[1]=#v9
    end

    if thread:isRunning() == false then
        thread:start(coco)

    end
    if thread2:isRunning() == false then
        thread2:start(coco)

    end
    if thread3:isRunning() == false then
        thread3:start(coco)
        
    end
    if thread4:isRunning() == false then
        thread4:start(coco)
        
    end
    if thread5:isRunning() == false then
        thread5:start(coco)
        
    end
    if thread6:isRunning() == false then
        thread6:start(coco)
        
    end
    if thread7:isRunning() == false then
        thread7:start(coco)
        
    end
    if thread8:isRunning() == false then
        thread8:start(coco)
        
    end
    if thread9:isRunning() == false then
        thread9:start(coco)

    end
    

end

function love.draw()
    love.graphics.print(_VERSION, 300, 10)
    love.graphics.printf( love.timer.getFPS(), 650, 10, 500, "left")

end

All my thread files look like this.
I know I could have looped though the the with a for loop.

Code: Select all

c = love.thread.getChannel("test")
var = 0
local coco = {}
local dd =10
for i=1, 100000 do

	var = var+1
	coco[i] = i  -- comment this out to see the difference.
end

c:push(coco)

[zorg]

Two things:
- All löve threads start their own Luajit virtual machine, meaning their own separate lua instance. These can't share lua types between each other without copying everything, so you passing a table of size 100 000 to 9 threads means you're duplicating the contents 9 times...
- Since you restart each thread when it finishes, but keep those i1,...,i9 tables around, and don't clear them, the calculated data will be pushed back to the main thread into those tables... well, actually no, into the v1...v9 variables, and even though those are overwritten with the new tables, since this is pretty fast, the older ones aren't getting collected fast enough; this keeps happening until memory runs out.

I'd suggest both assessing what you actually want to use threads for, and to use löve's ByteData objects if you want a shared memory region that you can access across threads. (Although that does come with potential risks in terms of data coherency - be aware what would read and write to it, and when)

[yal2du]

(100k new entries / thread ) * (4 bytes / entry) * 9 threads * 60 Hz * 2 ≃ 432 MB/s of new *fragmented* allocations, not even including other interthread communication overhead ... kind of curious how well other interpreted languages' allocators and garbage collectors hold up to that sort of pressure, if you are comparing to something else (e.g. a congruous script in python, javascript et al.)

[zorg]

technically speaking it's 8 bytes per entry (double precision floats are 8*8=64 bits) so it's more like 864 MB/s.

[pgimeno]

I think the OP may have successfully demonstrated a leak. I have to check in more depth, though.

The extreme conditions just help demonstrate it faster, but if there's a leak, it's a risk for any long-running application.

It happens with two threads too. I haven't been able to reproduce it with a single thread. The main-to-thread communication doesn't seem to play a role.

[toralord]

Two things:
- All löve threads start their own Luajit virtual machine, meaning their own separate lua instance. These can't share lua types between each other without copying everything, so you passing a table of size 100 000 to 9 threads means you're duplicating the contents 9 times...
- Since you restart each thread when it finishes, but keep those i1,...,i9 tables around, and don't clear them, the calculated data will be pushed back to the main thread into those tables... well, actually no, into the v1...v9 variables, and even though those are overwritten with the new tables, since this is pretty fast, the older ones aren't getting collected fast enough; this keeps happening until memory runs out.

I'd suggest both assessing what you actually want to use threads for, and to use löve's ByteData objects if you want a shared memory region that you can access across threads. (Although that does come with potential risks in terms of data coherency - be aware what would read and write to it, and when)

I figured it out love.thread.channel:push/and pop will cache all Variant data put through it until you clear it.
So, if you push 1 time it will cache the data and, then you push 4 more time it will cache those data adding to the 1 you already cached.
Basically have to manually garbage collect what you push and pop through the thread channel yourself.
Now, the data being cache won't hurt if run it once like in the thread example on the wiki page but,
in a game you might use a thread more then once like to load and save, AI for enemies in a turn base strategy etc.
all running in the background the cached data will add up causing a memory leak.
So, something like this.

Code: Select all

    local info = love.thread.getChannel( 'info' ):pop()
    -- Clrear cache data in the thread Channel to prevent memory leaks.
    love.thread.getChannel( 'info' ):clear()
    

I didn't use it earlier because on the wiki page of the cannel:clear() is reads

This function doesn't free up the memory used by a Channel, it needs to be garbage collected to release the memory.

but, that is exactly what it is doing. Freeing up memory by a channel to be garbage collected and release the memory.

edit--
I think my lua files got corrupted. I started a new files with the same code and it is working with out using cannel:clear() and it is working
just fine now. It is definitely corrupted lua files I dont know my VSCode somehow corrupted my lua files.