Prevent freezing while working with large files?

[typx]

Thanks alot, I really appreciate your help! Everything is working now. Sadly, it seems like Love isn't the right environment for this kind of tool. It's just super slow and after some minutes, it's freezing my computer At least I've learned a lot for future projects

[KayleMaster]

Can you share the .love file? I'm curious as to why it's slow because with luajit it's only 1.3x times slower than C.

[typx]

I can send it later when I'm at home. Imho it's the iterating of some of the files. The largest ones have up to 600k lines and it takes really long to read it line by line. I've tried getting the file as a filedata object and reading char by char until "\n" to get the lines but it's as slow as the other method.

[KayleMaster]

Well in that case, yeah. How much faster do you need ? It might be better to read the whole file in the thread without sending each line to the other thread and do the stuff you need to do with it there.

[typx]

That's how i did build it allready. The whole reading, analyzing, manipulating and writing stuff is run in the thread now. Problem is, the progressbar stuff is drawn in main so somehow I either need to send the lines or the progress calculation to main which both causes massive lags and freezes. Well, it's way better when using channel:supply() but it's taking hours then

I did build an earlier version of such a tool reading, manipulating and writing the same files in VB.net some months ago and everything was done in seconds. Here, a single medium-sized file alone takes longer

[zorg]

I still don't really get the issue to be honest; reading in from disk/ssd into ram line by line will always be slow as heck (doing it byte-by-byte is worse), just load it in all at once, or use a large enough buffer, like in the megabytes range; updating your progress bar from 0 to 100% in one step is not any less correct than sacrificing loading speed deliberately just so your users could look at a line being filled

Also, if you have to process the data further after you loaded it in, you can still do that, in RAM, which will be many times faster than the loadline-process loop.

By the way, you never really explained why you needed to read the file line-by-line; and "to have the progress bar graphic update neatly" is not a good reason ever.

[typx]

I've never said I need to read it line by line, also as I wrote earlier today, I've also tried working with the whole file as FileData which didn't boost the performance. Afaik, love.filesystem.lines and love.filesystem.read are the only options, according to the wiki.

Also, I don't do a 0 to 100 thing at my progressbar. It's 100 / NoOfFiles, added after every file so users can see what's happening. It doesn't have to be "shiny" but usable.

[KayleMaster]

It sounds like you can't send your progress properly.
EDIT: READ LAST CODE SNIPPET

Code: Select all

--thread.lua
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    channel.progress:pop()
    channel.progress:push(progressNumber)
    --read lines etc
  end
end
::done::

It would not cause freezes if we only kept one message in the queue. So we only pop/push from the thread. We do :peek from the main thread instead, which doesn't pop the message.

Code: Select all

--main.lua or wherever
local progressNumber = channelProgress:peek()

If even this causes slowdown, try sending the progress every 100 lines or so:

Code: Select all

local counter = 0
Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    counter = counter + 1
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    if counter == 100 then
    channel.progress:pop()
    channel.progress:push(progressNumber)
    counter = 0
    end
    --read lines etc
  end
end
::done::

EDIT: nevermind, just read you need the file count
Try this:

Code: Select all

Files = love.filesystem.getDirectoryItems("data")
for iFiles = 1, #Files do  
  channel.progress:pop()
  channel.progress:push(iFiles/#Files)
  for Line in love.filesystem.lines("data/" .. Files[iFiles]) do
    DoCancel = ChannelCancel:pop()
    if DoCancel then goto done end
    --read lines etc
  end
end
::done::

And then just do channelProgress:peek() from the main.lua
Obviously you need to make a channel for progress as well.

[pgimeno]

Strings in LuaJIT are slow to handle. Using string.byte tends to give much better performance than string.sub. Not sure if that's the issue here, but I thought it was worth noting.

[typx]

I guess the issue is working with the file itself. When using other languages, I just load the whole file into an array and save it after work is done. Guess using FileData = love.filesystem.read(File) doesn't do the trick with lua/love since it's the same speed als love.filesystem.lines