Prevent freezing while working with large files?

[typx]

Hey, I'm writing some dev tools for a future game. Currently I need to read approximately 200 text files line by line which have a size of 2 KB up to 7 MB. Everything's working fine, I just wonder if there's a way to prevent the ugly freezing while reading the files. I'm using imgui as an interface and it would be nice to make use of a progress bar while reading those files. I've tried love.thread and also coroutines but my knowledge stops there and I have no idea what to do

Code: Select all

function love.draw()
  --drawing stuff
  if imgui.Button("Read files") then
    if Working == false then
      Working = true
      ReadFiles()
    end
  end
  --drawing stuff
end

function ReadFiles()
  while Working do
    Files = love.filesystem.getDirectoryItems("data")
    for iFiles = 1, #Files do
      for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
        --read lines
      end
    end
    Working = false
  end
end

[KayleMaster]

Threads are the way to go, but it's a bit complicated. I'm on mobile now but maybe I can help later.

[KayleMaster]

Ok, here the zip file. I didn't make a .love but you can run this the normal way.
Also side note, the conf file is for 0.10.2 so if you're on 11 go ahead and copy the conf from here:
https://love2d.org/wiki/Config_Files
And change console = true (if you're on windows), and vsync = true just to limit the fps the easy way.

So now I'm gonna go through the important part of my code step by step:

Code: Select all

local thread -- Our thread object.
local receiveLine = love.thread.getChannel( "sendLine" )
local threadAction = love.thread.getChannel( "action" )


function love.load()
    thread = love.thread.newThread( "threadRead.lua" )
end

We create local variables for our thread and two of our channels. The sendLine channel stored in receiveLine will be used to receive the lines read from the files from the thread. The action thread is used to notify when the thread is over, although it really isn't necessary since we can just use thread:isRunning().

We load the thread code using newThread and the directory of the lua file.

Code: Select all

function love.update(dt)
    imgui.NewFrame()
    
    -- Make sure no errors occured.
    local error = thread:getError()
    assert( not error, error )

    if Working then
        local threadIsDone = threadAction:pop() or not thread:isRunning()
        if threadIsDone then Working = false end
        local received = receiveLine:pop()
        if received then
            print(received)
        end
    end

    miniTimer = miniTimer + 1
    if miniTimer == 20 then
        freezeNumber = freezeNumber + 1
        freezeNumber = freezeNumber % 100
    end
    miniTimer = miniTimer % 30
end

The two lines below the Make sure no errors occured are very important. They will notify you if something went wrong in the thread. Without those you wouldn't know. When I think about it, that code can be put in the 'if Working' block, since if it's not Working the thread wouldn't be running as well.

Code: Select all

local threadIsDone = threadAction:pop() or not thread:isRunning()
        if threadIsDone then Working = false end

This is self explanatory. The only change I would make was add an else and put the rest of the block inside it. But it doesn't matter.

Code: Select all

local received = receiveLine:pop()
        if received then
            print(received)
        end

We pop from the receiveLine channel (a channel is essentially a queue, a first in - first out data structure). If there's nothing, we'll have nil in received so we check for that.
The miniTimer code is to visualise if the game freezes while reading.

Code: Select all

function love.draw()
  --drawing stuff
  if imgui.Button("Read files") then
    if Working == false then
      Working = true
      thread:start()
    end
  end
  
  imgui.Render()
  love.graphics.print(freezeNumber)
  --drawing stuff
end

We start the thread using the method :start. You can find the methods for channels and threads here:
Channel
Thread

threadRead.lua

Code: Select all

require 'love.filesystem'

local channel = {}
channel.sendLine = love.thread.getChannel( "sendLine" )
channel.action = love.thread.getChannel( "action" )
print("Started reading")
local Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
    for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    --read lines
    channel.sendLine:supply(Line)
    end
end
channel.action:push(true)

We load the love.filesystem module because on a thread, only the love.thread module is loaded and if we need any love modules we can just require them like so. This time I decided to store the channels in a table, I find it more readable this way.
I use the :supply method to send the Line over to the main thread. The other method is push, but there's a catch - if you use push and the main thread is busy, the Lines will fill up the queue and use a lot of memory. This memory won't be freed and considering you're reading lots of files I figured supply would be better. What supply does is it sends a message to a thread Channel and wait for a thread to accept it as opposed to the push which just sends the message and continues the operation. Using push in this case won't freeze the thread waiting for the main thread to accept the message but may use a bit more memory.
At the end I send a simple boolean through the action channel to notify we're done. As I said earlier this isn't necessary since we can just use thread:isRunning() but I did it to show that you can have many channels in a thread.

Hope this helped, I'm pretty bad at writing tutorials but this should help you a little.

[typx]

Oh wow, thanks a lot for your effort. I'm not home until tomorrow, so I can't give it a try earlier.

[KayleMaster]

You're welcome. One more side note - I dunno if the thread dies properly, but I just read that the thread is killed when it returns. So just add return true or something at the end of the thread code in threadRead.

[typx]

So, I guess I got the most stuff now, but can you tell me how to send arguments to the thread? I know i can puth them into thread:start(Args) but I wasn't able to find out how to receive them inside the thread code. I tried making more channels, using channel:push() and channel:pop() which kind of worked, but it brought my computer to it's knees, nearly freezing windows for some minutes

[Nixola]

The arguments will be received as vararg. For example:
main.lua:

Code: Select all

thread = love.thread.newThread("thread.lua")
thread:start("foo", "bar", 4200)

thread.lua:

Code: Select all

local user, pass, port = ...
print(user, pass, port) -- this will print foo     bar     4200

[KayleMaster]

I knew I missed something!

If you're unsure how many arguments you're gonna send, you can also do:

Code: Select all

local arg_table = {...}
print(arg_table[1], arg_table[2])

[typx]

Okay, everything is working fine now expect some new thing i wanted to build in

Since the whole process of reading ~50MB of Textfiles line by line is super slow, I've tried to make use of a cancel button. I've tried to send an argument to the thread which should cancel the whole filereading. That's what I've tried:

Code: Select all

--main.lua & thread.lua
ChannelCancel = love.thread.getChannel("cancel")
DoCancel = false

--main.lua
function love.draw()
  --draw stuff
  if imgui.Button("Cancel") then
    DoCancel = true
    ChannelCancel:push(DoCancel)
  end
end

--thread.lua
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then break end
    --read lines etc
  end
end

Sadly, it's not working at all. When using print(DoCancel) its always false (sometimes even nil at first run with some earlier attempts). I've tried to put the push command into love.update() but that slowed my computer totally down.

I guess that's the last hurdle with all this new threading

[KayleMaster]

No, no, no. Why would you put :push into love.update. That would fill up the queue with the same messages 60 messages a second!
Anyways, you'd need to return, not break. Since you have 2 loops you only break from the outer one, the inner one is still working.
If you still have to do some stuff after cancel and return wouldn't work for you, you can also do:

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    --read lines etc
  end
end
::done::
--do more stuff here if you want 

If you want to do it with breaks, you can do it like so: (I think)

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then break end
    --read lines etc
  end  
  if DoCancel then break end
end

EDIT: the reason why pushing every tick worked is because it just skipped the inner loop. This is unefficient because you push messages and the inner loop removes them (pop). You could get away with doing :peek instead. But again, you'll still be going through the outer loop.