How to save and reload a very large amount of data?

[amorphia]

Hi all!

I need to be able to save and load a very large amount of data, because I want to be able to do in-game replays of games that have already been played.

So as I draw each object, I save the time, location, identity, etc, to a table, and when the game is over, I use bitser (https://github.com/gvx/bitser) to save the table. That works fine, even though the files are obviously large.

Then, in the next game, when I need to replay the recording, I load the saved file. The problem is, if the file is large (1Mb is OK, 3Mb is not) then the bitser.loadLoveFile() just crashes. Not a lua error message, a full on Windows "this program has stopped working" crash.

Is bitser not designed for large files? Is there a better way to do this? Any tips much appreciated.

Cheers,

Amorphia

[raidho36]

It's possible that bitser uses way to much memory internally, overloading Lua's capability. In the end, your computer memory can only store so much data at a time. The solution is to use some sort of streaming. Write data to disk frequently but in small packets and immediately reuse the memory to store next packet. During replay, only load few packets at a time, reuse memory from packets you already played back.

[zorg]

I'd wager that Robin knows better than to inflate 3 megabytes of data into over 2 gigabytes in-memory as to crash luaJIT...
3MB is not that big of an amount to be honest, maybe it's a bug in bitser, or your usage of it, can't tell without seeing code.

[amorphia]

Thanks a lot for the suggestions folks. If I'm going to have to write code for streaming, I might as well roll my own routines for all of this - I'd prefer to do neither. Here is minimal code which recreates the bug. Run it, wait a few seconds, press escape, run it again, press r. Advice to whether this is my bug or bitser's much appreciated!

Code: Select all

bitser = require 'bitser'

drawReg = {}
regIndex = 1

function love.load()
	love.filesystem.setIdentity('megaAttackGame');
end

function love.update(dt)
	if love.keyboard.isDown("escape") then
		success = bitser.dumpLoveFile('replay.dat', drawReg )
		love.event.quit()
	elseif love.keyboard.isDown("r") then
		replayData = bitser.loadLoveFile('replaySave2.dat')
	end
end

function love.draw(dt)
	for i=1,500 do
		registerVisible( 1, 1, "thing")
	end
end

function registerVisible( xPos, yPos, thingToDraw )
	drawReg[regIndex] = {
		timeStamp = os.clock(),
		x = xPos,
		y = yPos,
		t = thingToDraw
	}
	regIndex = regIndex + 1
	--Full version of function draws the thing as well
end

[zorg]

Code: Select all

success = bitser.dumpLoveFile('replay.dat', drawReg )

That function does not return any values, so success will always be nil.

Also, the two filename strings should be the same to get the error.

As for the actual error's reason... i'm not sure.
it works for 500 and 5000 iterations.
for 50 000 it goes over the deserialize_value function 550003 times, and returns the correct amount of items.
for 500 000, it flat out dies in the deserialize_value function.

My best bet, open an issue on github, that'd be the fastest way to get help regarding this.

Also, you don't want to detect keypresses for this kind of thing in update... it might run the function more than once (it happened with me since i held down the keys a bit long); you want to use the love.keypressed callback instead.

[Robin]

I'm looking into this, but I'm quite busy at the moment, so it might take a few days or longer before I get around to fixing this issue. There might be some inherent limitation we're running into, in which case I'll update the documentation.

[amorphia]

Thanks Robin, and everyone else who's been helping!

If it does turn out this isn't going to be a job for Bitser, I have made a plan B. I would output a file actually in .lua format, declaring a table, and then just require the file to re-import it. The file would be bigger than necessary as it would be text, but that's not a huge problem for my application. Is that a silly way to do it?

[zorg]

Thanks Robin, and everyone else who's been helping!

If it does turn out this isn't going to be a job for Bitser, I have made a plan B. I would output a file actually in .lua format, declaring a table, and then just require the file to re-import it. The file would be bigger than necessary as it would be text, but that's not a huge problem for my application. Is that a silly way to do it?

You can absolutely do that, something like (semi-pseudocode)

Code: Select all

function save(path, events)
local s = { "return {" }
for i,v in ipairs(events) do
-- You'd handle different types of vars differently, like saving "true" or "false" strings for bools, enclosing strings with ', etc...
table.insert(s,('%16.16g'):format(v)) -- for numbers, for example.
end
-- close off the table.
table.insert("}")
-- then write it out into a file, using table.concat or not; with will waste a bit of memory, without will maybe take a bit longer (writing things at once vs. everything sequentially, depends on buffers and stuff, not that important)
end

[alloyed]

I've also experienced crashes in bitser with large tables, but I couldn't reproduce it outside of my game so I just designed around it. in my case it didn't seem to be related to serialized file size: the test files I dumped before each crash were all over the place when it came to size, and maybe more importantly wouldn't crash if I made a program that just loaded the dumped files, and then serialized them back.

When it comes to streaming using bitser, the lazy approach I took was just to adopt a metaformat: when I read and write, I read and write chunks that represent a full bitser string. The first 64 bits are the length of the chunk, and then we read the full chunk using that and pass the resulting string to bitser. This works fine, although I haven't tested it against file corruption etc. I guess if it became a problem we can checksum each chunk.

[amorphia]

Great, thanks everyone!