Blob.lua - binary serialization library

[grump]

Can I pass the ctype information to the library somehow? I have the string definition, after all.

Not currently supported. My current plan for transparent cdata support would require user code for all serialized ctypes:

Code: Select all

local ffi = require('ffi')

ffi.cdef('typedef struct { double x, y; } teststruct_t')
local ctype = ffi.typeof('teststruct_t')

ffi.metatype(ctype, {
	__index = {
		__typename = 'teststruct_t',

		__serialize = function(self, writer)
			return writer:number(self.x):number(self.y)
		end,

		__deserialize = function(ctype, reader)
			return ctype(reader:number(), reader:number())
		end,
	}
})

This solution still has some drawbacks: anonymous struct types are not supported (since they have no type name), and custom code would be required for all ctypes. ctypes from libraries with a metatype of their own, e.g. brinevector, wouldn't work.

I can implement support for this, but it will take a while.

Edit: cdata serialization as described above is now supported. Example code.

The __serialize() and __deserialize() functions are only required for complex structs with pointers, and for cases when ffi.sizeof() is not sufficient to get the true size of the data.
The __typename field is required, except when calling :cdata() manually with a type name, in which case the metatype is not required at all (e.g. when using brinevector).

Code: Select all

-- brinevector example (no __typename, __serialize, or __deserialize are defined in brinevector}
-- this works
writer:cdata('brinevector', vector)
...
vector = reader:cdata()

-- but this is not possible, because no type information is available (and metatypes are immutable)
writer:write(vector)
-- or
writer:table({ vector }) -- ditto

[Xii]

And here I was thinking I had been rudely demanding. You really delivered

The __serialize() and __deserialize() functions are only required for complex structs with pointers,

So for plain old data structs, just need __typename ?

and for cases when ffi.sizeof() is not sufficient to get the true size of the data.

As I were experimenting with this before, I noticed that ffi.sizeof() returns a padded size for my test struct. Will that be an issue?

[Xii]

Error

BlobWriter.lua:141: bad argument #2 to 'copy' (cannot convert 'table' to 'const void *')

Traceback

[C]: in function 'copy'
BlobWriter.lua:141: in function 'string'
BlobWriter.lua:156: in function '_writeTaggedPair'
BlobWriter.lua:323: in function 'write'
main.lua:31: in function 'load'
[C]: in function 'xpcall'
[C]: in function 'xpcall'

Code: Select all

local ffi = require("ffi")
local writer, reader = require("BlobWriter"), require("BlobReader")

ffi.cdef[[
	typedef struct
	{
		uint8_t one;
		uint16_t two;
		uint32_t four;
		uint64_t eight;
	}
	test;
]]

local meta =
{
	__index =
	{
		__typename = 'test',
	}
}
actual = ffi.metatype(ffi.typeof('test'), meta)

local data = {}
data.yksi = {1, 2, 3, 4}
data.kaksi = actual()

local blob = writer()
blob:write(data)

[grump]

So for plain old data structs, just need __typename ?

Adding a __typename is sufficient for all data that is not a pointer and does not contain pointers. For stuff with pointers you need custom __serialize and __deserialize functions to write/reconstruct the data manually. Otherwise only the pointer's addresses would be serialized.

As I were experimenting with this before, I noticed that ffi.sizeof() returns a padded size for my test struct. Will that be an issue?

It can be an issue if the serialized data needs to be portable between different architectures and machine word sizes. In cases where portability is required, using #pragma pack() in the cdef call makes sure consistent padding is applied.

The bug you've encountered is now fixed. I forgot to add a test case for cdata in hash tables and of course there was a small mistake in that code.

[Xii]

Awesome! And beautiful! Look at how simple and succint this is thanks to your diligent efforts:

Code: Select all

local ffi = require("ffi")
local writer, reader = require("BlobWriter"), require("BlobReader")

function define_struct(name, memberstr)
	ffi.cdef("#pragma pack(1)\ntypedef struct{"..memberstr.."}"..name..";")
	local meta =
	{
		__index =
		{
			__typename = name,
		}
	}
	return ffi.metatype(ffi.typeof(name), meta)
end

actual = define_struct("test",
[[
	uint8_t one;
	uint8_t oneb;
	uint8_t onec;
	uint16_t two;
	uint32_t four;
	uint64_t eight;
]])

local data = {}
data.foo = {1, 2, 3, 4}
data.bar = actual()

local blob = writer()
blob:write(data)
assert(love.filesystem.write("test", blob:tostring()))

Don't forget to bump your copyright up to 2021

[yal2du]

Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.

[GVovkiv]

Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.

https://github.com/megagrump/moonblob ?

[yal2du]

Is this library still available (somewhere)? The url github.com/megagrump/blob goes to 404.

https://github.com/megagrump/moonblob ?

Yes, thanks. I presumed it was moonscript but the lua source is there as well.

[grump]

Yeah it was originally written in vanilla Lua and later rewritten in moonscript and renamed. The Lua code in the repo is there for convenience so you don't have to compile it yourself.

I've changed the link in the OP, thanks for the hint.