bitser: faster, harder, better serialization with LuaJIT

[Robin]

Oops I did it again
I played with serialization, got lost in the game
Oh baby, baby

bitser is the ultimate serialization library for LÖVE:

• Uses LuaJIT to be superfast and superefficient
• Just like Lady and binser, it supports registering resources (like Image, Source, anything) and classes (MiddleClass, SECL, hump.class, Slither)
• It's secure: you can use it to serialize over teh interwebs
• LÖVE bonus: you can load serialized values from FileData objects as well as from strings, which means loading gets even faster
• Seriously, try it out, if you run the repo with LÖVE you get a benchmark with nice charts that you can compare if you download other serialization libraries to it.

[qaisjp]

Nice work! What kind of format does it serialize to?

[Robin]

It uses a custom binary format that is optimized for size and writing time.

Here's an overview of each initial byte:

Code: Select all

0 * * * * * * * -> small int
1 0 * * * * * * -> small ref
1 1 0 * * * * * -> small string    [data:byte * rest]
1 1 1 0 * * * * -> small res       [label:byte * rest]
1 1 1 1 0 0 0 0 -> table           [len:number] [value:any] * len [keylen:number] ([key:any] [value:any]) * keylen
1 1 1 1 0 0 0 1 -> resource        [label:string]
1 1 1 1 0 0 1 0 -> instance        [class:string] [data:table]
1 1 1 1 0 0 1 1 -> reference       [idx:number]
1 1 1 1 0 1 0 0 -> string          [len:number] [data:byte * len]
1 1 1 1 0 1 0 1 -> long            [data:byte * 4]
1 1 1 1 0 1 1 0 -> float           [data:byte * 8]
1 1 1 1 0 1 1 1 -> nil
1 1 1 1 1 0 0 0 -> false
1 1 1 1 1 0 0 1 -> true
1 1 1 1 1 0 1 0 -> short           [data:byte * 2]
1 1 1 1 1 0 1 1 -> N/A
1 1 1 1 1 1 0 0 -> N/A
1 1 1 1 1 1 0 1 -> N/A
1 1 1 1 1 1 1 0 -> N/A
1 1 1 1 1 1 1 1 -> N/A

Type bytes 251-255 are reserved for future extension.

[prixt]

Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.

[pgimeno]

• It's secure: you can use it to serialize over teh interwebs

Yay! Security is the very reason I've used a JSON library instead of any of the other serializers out there, in my only project so far (which didn't use self-referential tables or any other stuff unsupported by JSON). Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.

The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.

However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.

[monolifed]

does small int work like variable int in protobuf (uses less bytes for small integers)

[airstruck]

Each and every serializer I've found that was able to save native Lua data in textual form, loaded the data through loadstring.

You could use setfenv with loadstring, it should be reasonably secure.

Code: Select all

local f = loadstring(serialized)
setfenv(f, {})
local deserialized = f() 

I don't think it could do any real damage sandboxed like that, although it could try to be annoying by hanging the game or something (someone please correct me if I'm wrong).

[Robin]

Is it possible to register custom classes that don't use libraries you mentioned? I prefer to create classes on a case by case basis, so don't use outside libraries.

It depends on how exactly you use it. Bitser has to heuristically detect how the class works and how to construct instances when loading. If the class is the metatable of instances, for example, it would work no problem, because bitser would think you're using hump.class. When registering a class you can also give a classkey and deserializer:

Code: Select all

bitser.registerClass('YourClass', YourClass, 'classkey', function(instance, class) return --[[make instance an instance of class]] end)

The fact that it outputs to binary only is a problem for me to adopt it, though. I'm not a big fan of obscurity in savegames in most situations. One should not need to use a savegame editor just to make a few changes.

That's a good point. That's maybe a good idea for another library, that maximizes readability at the expense of size, speed and security. Originally I wrote Lady for that, but Lady is not very readable, making hand-editing it less fun.

However, I like the idea for complex games where there's a lot of data and storing it in a compact way is best.

Yeah, that's more the intended use case.

does small int work like variable int in protobuf (uses less bytes for small integers)

Yes. Integers from -27 to 100 are stored in the type byte.

[bakpakin]

Wow. That is quality.

For luajit needs, binser might be obsolete. All that work I put into making sure numbers serialized correctly, and you go and just use ffi to copy memory.

The links for other libraries for the benchmark are broken, though. Replacing "github.com" in the urls with "raw.githubusercontent.com" should do the trick, I think.

Great work. This looks like it has applications outside of LOVE. I might use this with luvit.

[Robin]

Thanks!

Huh, the links worked for me. Maybe I should replace them then.