Page 3 of 3
Re: Prevent freezing while working with large files?
Posted: Wed Jun 20, 2018 9:27 am
by KayleMaster
You can still do that in C, using FFI, if you absolutely need to. I don't think loading a 600k line file into a contiguous array would work well tho.
Re: Prevent freezing while working with large files?
Posted: Wed Jun 20, 2018 9:48 am
by typx
Well, at least in VB.net I do kinda the same, using the same files to read.
Code: Select all
Dim DataFiles() As String = IO.Directory.GetFiles("data", "*.sav")
For iFiles = 0 To DataFiles.Length - 1
Dim Lines() As String = IO.File.ReadAllLines(DataFiles(iFiles))
For iLines = 0 To Lines.Length - 1
Line = Lines(iLines)
Next
Next
This iterates all files in 4 seconds, while the love attempt takes several minutes for 1MB+ files, even without any graphical extras.
Re: Prevent freezing while working with large files?
Posted: Wed Jun 20, 2018 9:57 am
by KayleMaster
I mean, I can optimize it if you need to. But I'd need your project folder and data to do that.
Anyways, Love2D isn't really meant for this, obviously it's a gamedev framework, but I don't think it's that slow.
Re: Prevent freezing while working with large files?
Posted: Wed Jun 20, 2018 10:12 am
by typx
Yeah, I know it's not meant for this. But I try to solve tasks I think are relatively simple with lua/love to get used to the framework, imgui design and lua in general.
A friend of mine is working on a machine learning engine and I will provide dev tools and ui frontend for a game using it later (or maybe someday
), so I try to do as much in lua/love as possible, even if it's a bit out of the ordinary.
Sadly, I can't post the data files here, cause there's some licensing involved (they're from a game we're modding) but I could PM you if you're interested.
Re: Prevent freezing while working with large files?
Posted: Wed Jun 20, 2018 10:41 am
by grump
typx wrote: ↑Wed Jun 20, 2018 9:48 am
This iterates all files in 4 seconds, while the love attempt takes several minutes for 1MB+ files, even without any graphical extras.
You must be running this on a potato or you're doing something severely wrong.
Code: Select all
local start = love.timer.getTime()
local lines = {}
for i = 1, 600000 do lines[i] = ('x'):rep(100) end
love.filesystem.write('test.txt', table.concat(lines, '\n'))
lines = {}
for line in love.filesystem.lines('test.txt') do
lines[#lines + 1] = line
end
print(#lines, "lines:", love.timer.getTime() - start, "seconds")
Output:
Code: Select all
600000 lines: 0.8547670400003 seconds
Generating a table of 600k lines with 100 chars each, joining them, writing them to a file, reading them back in and put them in a table takes less than one second for me. That's 57 MB of data. Granted, the data is still in the filesystem cache when reading it back.
Are you using LÖVE 11? 0.10.2 and below had a bug in love.filesystem.lines that made it perform extremely poorly when reading from a love file.
Re: Prevent freezing while working with large files?
Posted: Wed Jun 20, 2018 10:47 am
by typx
Ah, yeah I'm using 0.10.2, meh
Will give it a try with 11.1 tomorrow.
Re: Prevent freezing while working with large files?
Posted: Sat Jun 23, 2018 10:55 pm
by typx
Updating love helped a lot, reading just takes several seconds now
Sadly, when I try to manipulate and rewrite the files now, it's getting super slow again
love.filesystem.append single lines and also love.filesystem.write a huge string variable are unusable. I know writing to a table and just saving as lua is the way to go, but since we're modding an existing game which expects textfiles, that's not an option.
Re: Prevent freezing while working with large files?
Posted: Sun Jun 24, 2018 8:04 am
by bartbes
Can't you just get an actual file handle in append mode and keep appending that way? Sure sounds more efficient than repeatedly opening and closing the same file.
Re: Prevent freezing while working with large files?
Posted: Sun Jun 24, 2018 11:28 pm
by typx
Oh yeah, that did speed up everything a lot. Thanks to all of you