Questions on Saving data aka serializing table [SOLVED]

[Muris]

Hello and sorry for asking something trivial but I am still quite new to LUA

I tried couple of the lua serializing tools to serialize my object: https://github.com/gvx/Smallfolk and smallfolk failed to deserialize on "exp" = 0, it didn't like 0 as a number.

Second one I tried was https://love2d.org/wiki/Tserialwhich for some reason couldn't deserialize the stuff it serialized.

The third one I tried actually works just fine, but it returns unsafe lua-file: https://github.com/gvx/Ser
Basically the output is something as following in ser, where it just has a huge list of characters and their stats:

Code: Select all

_[36] = {"0", "300000"}

_[17] = {classType = "Ranger", important = true, expGain = 50, alias = "Peter", npc = false, spi = 77, vit = 155, lvl = 101, str = 112, skills = _[36], img = "8", int = 99, exp = 0, luc = 146, speed = 5, name = "Peter", dex = 230}

_[1] = {["4"] = _[2], test2 = _[3], ["5"] = _[4], test7 = _[5], test4 = _[6], test9 = _[7], test6 = _[8], ["7"] = _[9], test8 = _[10], ["1"] = _[11], ["6"] = _[12], test1 = _[13], test5 = _[14], ["3"] = _[15], test3 = _[16], ["2"] = _[17], test10 = _[18], test12 = _[19], test11 = _[20]}

return {units = _[1]}

So the units just have an id, and which refers to the unit, which has a list of skills on their disposal. I used strings for identifiers, since I was first using JSON to describe my character data.

So the way I am actually loading is by using loadstring, which could do a lot of unwanted stuff. So I was thinking if I run something like:

Code: Select all

local brackets = serString:find("[()]") 	
if not brackets then
  local f = assert(loadstring( serString ))
  local saveState = f()
  ... do my stuff
end

Now this leaves me to the question: If I take of brackets (), and make sure that I do not use brackets inside any of my variables, would it lead to complete safe code? Like can anyone do anything that can break stuff with the reserved words such as return, end etc? The reason why I am trying to remove () is to detect function declarations / function calls, although someone can still overwrite global function names I think like love.draw = nil

I am able to accept that the game crashes if the savefile is incorrect, but not stuff like delete files from harddrive by modifying the save file etc. Also since I am kinda trying to get this working in android at the same time, I think that I should be a bit more caution about these things.

[kikito]

You can always put your loadstring / ser code in a sandbox with sandbox.lua:

Code: Select all

local sandbox = require 'sandbox'
...
local saveState = assert(sandbox.run(serString))
...

sandbox.lua will prevent nasty things from being executable inside serString. The most harmful thing they will be able to do is block your game (with an infinite loop - we can prevent that in Lua 5.1, but not in LuaJIT).

[Muris]

You can always put your loadstring / ser code in a sandbox with sandbox.lua:

Code: Select all

local sandbox = require 'sandbox'
...
local saveState = assert(sandbox.run(serString))
...

sandbox.lua will prevent nasty things from being executable inside serString. The most harmful thing they will be able to do is block your game (with an infinite loop - we can prevent that in Lua 5.1, but not in LuaJIT).

Thanks, this definitely does seem to cover the case of changing global values.

I am still leaving the checking for brackets '(' and ')' for just in case, although it is probably unneccessary. I am just thinking about some odd case where someone defines a function in one of the table values which then changes the global values somehow when being called in some random case, although it shouldn't happen since everything should be either tables / strings or numbers, and thus it should always throw an exception when handling a function as other type of field. Maybe adding some validation for the data would cover most of this.

[kikito]

Even if they define a function inside loadstring, it won't be able to modify any global variable. This is due to the way the sandbox is built - functions declared in the loadstring will be inside a "protected scope", not in the global one.

[Muris]

Even if they define a function inside loadstring, it won't be able to modify any global variable. This is due to the way the sandbox is built - functions declared in the loadstring will be inside a "protected scope", not in the global one.

This is indeed quite nice, just tested it with following line of code:

Code: Select all

sandbox.run( "return function() print(\"test\") end")()

And it really is like you said, the whole global space is unreachable from sandboxed code, it cannot find print.

[Robin]

In the case you cannot trust the source of the data, there is actually another library made especially for this purpose: Smallfolk. It's output also tends to be smaller than Ser's, and you use smallfolk.loads(string) instead of loadstring(string)().

[Muris]

In the case you cannot trust the source of the data, there is actually another library made especially for this purpose: Smallfolk. It's output also tends to be smaller than Ser's, and you use smallfolk.loads(string) instead of loadstring(string)().

smallfolk wasnt able to deserialize a case with having a number 0 as a value. I had an object with exp = 0, and the deserializer threw exception on deserializing the string that it serialized.

Here is a code that it fails to work with:

Code: Select all

local testcase = smallfolk.dumps({ a = 0 })
print( smallfolk.loads(testcase).a )

Also 0. something doesnt work, it fails with:

Code: Select all

local testcase = smallfolk.dumps({ a = 0.12 })
print( smallfolk.loads(testcase).a )

And same with negative -0.somehting:

Code: Select all

local testcase = smallfolk.dumps({ a = -0.15 })
print( smallfolk.loads(testcase).a )

Also for some reason the object that I tried to serialize with TSerialize also failed to deserialize it its own serialization

edit added a testcase that the serializer cannot complete.
edit2: added another case that the deserializer doesnt work with

[Robin]

That is a serious bug that I'll fix right away. Can't believe I didn't test for 0!

EDIT: fix on github!

[Muris]

That is a serious bug that I'll fix right away. Can't believe I didn't test for 0!

EDIT: fix on github!

Nice thank you for the fast fix, which seems to also work with the 0. something aswell. Now it seems to be able to deserialize the character stats. Also it seems to be able to distinguish 0 and "0", so I think I won't have any other problems on using it.

[Robin]

Good to hear! If you run into any further problems, using either Smallfolk or Ser, I would appreciate you contact me, either directly, or by making an issue on the Github repository (I think you need a Github account for that), that way I can fix the problem as soon as possible.