Hi all!
I need to be able to save and load a very large amount of data, because I want to be able to do in-game replays of games that have already been played.
So as I draw each object, I save the time, location, identity, etc, to a table, and when the game is over, I use bitser (https://github.com/gvx/bitser) to save the table. That works fine, even though the files are obviously large.
Then, in the next game, when I need to replay the recording, I load the saved file. The problem is, if the file is large (1Mb is OK, 3Mb is not) then the bitser.loadLoveFile() just crashes. Not a lua error message, a full on Windows "this program has stopped working" crash.
Is bitser not designed for large files? Is there a better way to do this? Any tips much appreciated.
Cheers,
Amorphia
How to save and reload a very large amount of data?
Forum rules
Before you make a thread asking for help, read this.
Before you make a thread asking for help, read this.
Re: How to save and reload a very large amount of data?
It's possible that bitser uses way to much memory internally, overloading Lua's capability. In the end, your computer memory can only store so much data at a time. The solution is to use some sort of streaming. Write data to disk frequently but in small packets and immediately reuse the memory to store next packet. During replay, only load few packets at a time, reuse memory from packets you already played back.
- zorg
- Party member
- Posts: 3468
- Joined: Thu Dec 13, 2012 2:55 pm
- Location: Absurdistan, Hungary
- Contact:
Re: How to save and reload a very large amount of data?
I'd wager that Robin knows better than to inflate 3 megabytes of data into over 2 gigabytes in-memory as to crash luaJIT...
3MB is not that big of an amount to be honest, maybe it's a bug in bitser, or your usage of it, can't tell without seeing code.
3MB is not that big of an amount to be honest, maybe it's a bug in bitser, or your usage of it, can't tell without seeing code.
Me and my stuff True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.
Re: How to save and reload a very large amount of data?
Thanks a lot for the suggestions folks. If I'm going to have to write code for streaming, I might as well roll my own routines for all of this - I'd prefer to do neither. Here is minimal code which recreates the bug. Run it, wait a few seconds, press escape, run it again, press r. Advice to whether this is my bug or bitser's much appreciated!
Code: Select all
bitser = require 'bitser'
drawReg = {}
regIndex = 1
function love.load()
love.filesystem.setIdentity('megaAttackGame');
end
function love.update(dt)
if love.keyboard.isDown("escape") then
success = bitser.dumpLoveFile('replay.dat', drawReg )
love.event.quit()
elseif love.keyboard.isDown("r") then
replayData = bitser.loadLoveFile('replaySave2.dat')
end
end
function love.draw(dt)
for i=1,500 do
registerVisible( 1, 1, "thing")
end
end
function registerVisible( xPos, yPos, thingToDraw )
drawReg[regIndex] = {
timeStamp = os.clock(),
x = xPos,
y = yPos,
t = thingToDraw
}
regIndex = regIndex + 1
--Full version of function draws the thing as well
end
- zorg
- Party member
- Posts: 3468
- Joined: Thu Dec 13, 2012 2:55 pm
- Location: Absurdistan, Hungary
- Contact:
Re: How to save and reload a very large amount of data?
Code: Select all
success = bitser.dumpLoveFile('replay.dat', drawReg )
Also, the two filename strings should be the same to get the error.
As for the actual error's reason... i'm not sure.
it works for 500 and 5000 iterations.
for 50 000 it goes over the deserialize_value function 550003 times, and returns the correct amount of items.
for 500 000, it flat out dies in the deserialize_value function.
My best bet, open an issue on github, that'd be the fastest way to get help regarding this.
Also, you don't want to detect keypresses for this kind of thing in update... it might run the function more than once (it happened with me since i held down the keys a bit long); you want to use the love.keypressed callback instead.
Me and my stuff True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.
- Robin
- The Omniscient
- Posts: 6506
- Joined: Fri Feb 20, 2009 4:29 pm
- Location: The Netherlands
- Contact:
Re: How to save and reload a very large amount of data?
I'm looking into this, but I'm quite busy at the moment, so it might take a few days or longer before I get around to fixing this issue. There might be some inherent limitation we're running into, in which case I'll update the documentation.
Help us help you: attach a .love.
Re: How to save and reload a very large amount of data?
Thanks Robin, and everyone else who's been helping!
If it does turn out this isn't going to be a job for Bitser, I have made a plan B. I would output a file actually in .lua format, declaring a table, and then just require the file to re-import it. The file would be bigger than necessary as it would be text, but that's not a huge problem for my application. Is that a silly way to do it?
If it does turn out this isn't going to be a job for Bitser, I have made a plan B. I would output a file actually in .lua format, declaring a table, and then just require the file to re-import it. The file would be bigger than necessary as it would be text, but that's not a huge problem for my application. Is that a silly way to do it?
- zorg
- Party member
- Posts: 3468
- Joined: Thu Dec 13, 2012 2:55 pm
- Location: Absurdistan, Hungary
- Contact:
Re: How to save and reload a very large amount of data?
You can absolutely do that, something like (semi-pseudocode)amorphia wrote: ↑Mon May 29, 2017 4:02 pm Thanks Robin, and everyone else who's been helping!
If it does turn out this isn't going to be a job for Bitser, I have made a plan B. I would output a file actually in .lua format, declaring a table, and then just require the file to re-import it. The file would be bigger than necessary as it would be text, but that's not a huge problem for my application. Is that a silly way to do it?
Code: Select all
function save(path, events)
local s = { "return {" }
for i,v in ipairs(events) do
-- You'd handle different types of vars differently, like saving "true" or "false" strings for bools, enclosing strings with ', etc...
table.insert(s,('%16.16g'):format(v)) -- for numbers, for example.
end
-- close off the table.
table.insert("}")
-- then write it out into a file, using table.concat or not; with will waste a bit of memory, without will maybe take a bit longer (writing things at once vs. everything sequentially, depends on buffers and stuff, not that important)
end
Me and my stuff True Neutral Aspirant. Why, yes, i do indeed enjoy sarcastically correcting others when they make the most blatant of spelling mistakes. No bullying or trolling the innocent tho.
Re: How to save and reload a very large amount of data?
I've also experienced crashes in bitser with large tables, but I couldn't reproduce it outside of my game so I just designed around it. in my case it didn't seem to be related to serialized file size: the test files I dumped before each crash were all over the place when it came to size, and maybe more importantly wouldn't crash if I made a program that just loaded the dumped files, and then serialized them back.
When it comes to streaming using bitser, the lazy approach I took was just to adopt a metaformat: when I read and write, I read and write chunks that represent a full bitser string. The first 64 bits are the length of the chunk, and then we read the full chunk using that and pass the resulting string to bitser. This works fine, although I haven't tested it against file corruption etc. I guess if it became a problem we can checksum each chunk.
When it comes to streaming using bitser, the lazy approach I took was just to adopt a metaformat: when I read and write, I read and write chunks that represent a full bitser string. The first 64 bits are the length of the chunk, and then we read the full chunk using that and pass the resulting string to bitser. This works fine, although I haven't tested it against file corruption etc. I guess if it became a problem we can checksum each chunk.
Re: How to save and reload a very large amount of data?
Great, thanks everyone!
Who is online
Users browsing this forum: Ahrefs [Bot], Bing [Bot], Google [Bot] and 3 guests