Best way of serializing data for transmission over network interface

[steven777]

Hello iv been doing some research and am wondering whats the preferred method of making a secure network for your games.

iv seen Tserial being used but i hear its not safe on the server side. ??

am i wrong about this or does anyone have a code snip-it that could show me the proper use of this over a udp or tpc network ( basic code i dont need non of you guys writing pages XD)

i see that Tserial also has an unpack function, I am not sure if this is new compared to some other libs people have made to work around security issues with unpacking a table.



currently my game just works on sending tables with low data. just to connect players to the game and start basic movment.

I am worried that if I make a game that I will be hacked and run a risk of losing everything that I wil have worked on in the end... are my worries unwarranted??


thank you for the help guys, i really appreciate this community.

edit:

or should I just send strings and unpack the strings into tables... this method looks increasingly hard and have no idea about match patterns.

if someone could help me better understand what it is that I need to be doing please share.

[steven777]

so ti seems that it is loadstring that gets you into trouble it seems... because a malicious client could send you function calls and screw up everything throw errors. what is byte code and why is this an issue ?

does this mean if i make my game around this I am surely to get hacked? i need to speak with someone..

[Plu]

I don't know what the "best way" is, but the general principles of user input apply here:

1) assume everything sent is malicious
2) validate all possible input
3) only accept the input that you expect

The basic idea would probably be to encode only basic data (numbers and strings in tables), and upon receiving them, to unpack the data, validate everything and only if the data is indeed only numbers and strings, feeding it to the rest of the program.

(I don't have any examples here, though)

[steven777]

so really what i need is an example of some ways to pack and unpack tables correctly. I am not trying to send functions calls over the net. just data and strings for player checking and such.

Does anyone here know how to pack and unpack strings or tables properly ?

[pgimeno]

bitser may be of help; I also recently found Smallfolk which is the same idea but using ASCII instead of binary. It's the only serialization library I know that generates ASCII and deserializes without executing. Both by forum member Robin.

If compactness is a goal, go for bitser; if readability, Smallfolk. I wish I had known it before. I used JSON because of security concerns.

[steven777]

i am trying to learn how to build this system on my own with the addition of someones crap library that gets me a little closer to the goal bt not quite there..

all i need to know is how to pass strings, or tables threw a udp or tcp network.

i know i have to pack the table and send it. im trying to do so without the need of libratys or match codes and strings. i just need a function to pack and un pack the table. before and after sending..



so im just looking for a work around for all of these match codes and strings in this example...

https://love2d.org/wiki/Tutorial:Networking_with_UDP

[pgimeno]

Well, to pass a string, you can just send it, but you can't pass a table without serializing it. Serializing is converting the data (table, number, string, whatever) into a string or some other kind of contiguous, 1-D stream (including a network transmission). If you feel like writing a serialization and deserialization library, feel free. Otherwise you'll have to use one already made like the ones I've suggested. I've given you the best ones I know if your concern is security.

[zorg]

all i need to know is how to pass strings, or tables threw a udp or tcp network.

As pgimeno said above, you can throw data that aren't strings already (or even those) into a serializer, or heck, just push them through tostring... though it will most certainly not be the most compressed output you'd want to send to other places.
Actually sending those is of course dependent on what path you choose to follow. (As in, you'd need to read up and understand the send method of the object created by socket.udp, or alternatively, enet's peer:send; not the hardest parts of all this...)

Also, robin's libs are also heavily optimized, not really in any category one could define as "crap" per se.

That said, networks ain't UDP or TCP, those are protocols, or, in smaller words, the means which you can pass data from one "location" to another.
I'd also advocate using [wiki]lua-enet[/wiki] instead of (UDP) sockets, maybe it's a bit more clearer than the other option, though that's just my preference, you can feel free to disregard this. (Then again, don't use TCP for games.)

[Pangit]

Zorg; that Glenn Fiedler article is great - it answered many of the questions I had about using UDP in games. Thanks.

[steven777]

i checked out his Lib an while i must say it isnt crap, but i have no idea how it actually works or serializes a table into a string...


if someone could describe how to serialize data. I would like to write my own lib for the game. My game is not open scource but the foundation of it will be.


in the UDP networking wiki for love 2D there are several lines i questions about not just in the lib.






client origonal wiki code:

Code: Select all

- to start with, we need to require the 'socket' lib (which is compiled
-- into love). socket provides low-level networking features.
local socket = require "socket"
 
-- the address and port of the server
local address, port = "localhost", 12345
 
local entity -- entity is what we'll be controlling
local updaterate = 0.1 -- how long to wait, in seconds, before requesting an update
 
local world = {} -- the empty world-state
local t
 
-- love.load, hopefully you are familiar with it from the callbacks tutorial
function love.load()
 
	-- first up, we need a udp socket, from which we'll do all
	-- out networking.
	udp = socket.udp()
 
	-- normally socket reads block until they have data, or a
	-- certain amout of time passes.
	-- that doesn't suit us, so we tell it not to do that by setting the 
	-- 'timeout' to zero
	udp:settimeout(0)
 
	-- unlike the server, we'll just be talking to the one machine, 
	-- so we'll "connect" this socket to the server's address and port
	-- using setpeername.
	--
	-- [NOTE: UDP is actually connectionless, this is purely a convenience
	-- provided by the socket library, it doesn't actually change the 
	--'bits on the wire', and in-fact we can change / remove this at any time.]
	udp:setpeername(address, port)
 
	-- seed the random number generator, so we don't just get the
	-- same numbers each time.
	math.randomseed(os.time()) 
 
	-- entity will be what we'll be controlling, for the sake of this
	-- tutorial its just a number, but it'll do.
	-- we'll just use random to give us a reasonably unique identity for little effort.
	--
	-- [NOTE: random isn't actually a very good way of doing this, but the
	-- "correct" ways are beyond the scope of this article. the *simplest* 
	-- is just an auto-count, they get a *lot* more fancy from there on in]
 
	entity = tostring(math.random(99999))
 
	-- Here we do our first bit of actual networking:
	-- we set up a string containing the data we want to send (using 'string.format')
	-- and then send it using 'udp.send'. since we used 'setpeername' earlier
	-- we don't even have to specify where to send it.
	--
	-- thats...it, really. the rest of this is just putting this context and practical use.
	local dg = string.format("%s %s %d %d", entity, 'at', 320, 240)
	udp:send(dg) -- the magic line in question.
 
	-- t is just a variable we use to help us with the update rate in love.update.
	t = 0 -- (re)set t to 0
end
 
-- love.update, hopefully you are familiar with it from the callbacks tutorial
function love.update(deltatime)
 
	t = t + deltatime -- increase t by the deltatime
 
	-- its *very easy* to completely saturate a network connection if you
	-- aren't careful with the packets we send (or request!), we hedge
	-- our chances by limiting how often we send (and request) updates.
	-- 
	-- for the record, ten times a second is considered good for most normal
	-- games (including many MMOs), and you shouldn't ever really need more 
	-- than 30 updates a second, even for fast-paced games.
	if t > updaterate then
		-- we could send updates for every little move, but we consolidate
		-- the last update-worth here into a single packet, drastically reducing
		-- our bandwidth use.
		local x, y = 0, 0
		if love.keyboard.isDown('up') then 	y=y-(20*t) end
		if love.keyboard.isDown('down') then 	y=y+(20*t) end
		if love.keyboard.isDown('left') then 	x=x-(20*t) end
		if love.keyboard.isDown('right') then 	x=x+(20*t) end
 
 
		-- again, we prepare a packet *payload* using string.format, 
		-- then send it on its way with udp:send
		-- this one is the move update mentioned above
		local dg = string.format("%s %s %f %f", entity, 'move', x, y)
		udp:send(dg)	
 
		-- and again! this is a require that the server send us an update for
		--  the world state
		--
		-- [NOTE: in most designs you don't request world-state updates, you
		-- just get them sent to you periodically. theres various reasons for
		-- this, but theres one *BIG* one you will have to solemnly take note
		-- of: 'anti-griefing'. World-updates are probably one of biggest things
		-- the average game-server will pump out on a regular basis, and greifing
		-- with forged update requests would be simple effective. so they just
		-- don't support update requests, instead giving them out when they feel
		-- its appropriate]
		local dg = string.format("%s %s $", entity, 'update')
		udp:send(dg)
 
		t=t-updaterate -- set t for the next round
	end
 
 
	-- there could well be more than one message waiting for us, so we'll
	-- loop until we run out!
	repeat
		-- and here is something new, the much anticipated other end of udp:send!
		-- receive return a waiting packet (or nil, and an error message).
		-- data is a string, the payload of the far-end's send. we can deal with it
		-- the same ways we could deal with any other string in lua (needless to 
		-- say, getting familiar with lua's string handling functions is a must.
		data, msg = udp:receive()
 
		if data then -- you remember, right? that all values in lua evaluate as true, save nil and false?
 
			-- match is our freind here, its part of string.*, and data is
			-- (or should be!) a string. that funky set of characters bares some 
			-- explanation, though.
			-- (need summary of patterns, and link to section 5.4.1)
			ent, cmd, parms = data:match("^(%S*) (%S*) (.*)")
			if cmd == 'at' then
				-- more patterns, this time with sets, and more length selectors!
				local x, y = parms:match("^(%-?[%d.e]*) (%-?[%d.e]*)$")
				assert(x and y) -- validation is better, but asserts will serve.
 
				-- don't forget, even if you matched a "number", the result is still a string!
				-- thankfully conversion is easy in lua.
				x, y = tonumber(x), tonumber(y)
				-- and finally we stash it away
				world[ent] = {x=x, y=y}
			else
				-- this case shouldn't trigger often, but its always a good idea
				-- to check (and log!) any unexpected messages and events.
				-- it can help you find bugs in your code...or people trying to hack the server.
				-- never forget, you can not trust the client!
				print("unrecognised command:", cmd)
			end
 
		-- if data was nil, then msg will contain a short description of the
		-- problem (which are also error id...).
		-- the most common will be 'timeout', since we settimeout() to zero,
		-- anytime there isn't data *waiting* for us, it'll timeout.
		--
		-- but we should check to see if its a *different* error, and act accordingly.
		-- in this case we don't even try to save ourselves, we just error out.
		elseif msg ~= 'timeout' then 
			error("Network error: "..tostring(msg))
		end
	until not data 
 
end
 
-- love.draw, hopefully you are familiar with it from the callbacks tutorial
function love.draw()
	-- pretty simple, we just loop over the world table, and print the
	-- name (key) of everything in their, at its own stored co-ords.
	for k, v in pairs(world) do
		love.graphics.print(k, v.x, v.y)
	end
end
 
-- And thats the end of the udp client example.

the lines i questions about are bellow:

Code: Select all

		local dg = string.format("%s %s %f %f", entity, 'move', x, y)
		udp:send(dg)

i understand the UDP and TCP stuff.

its the( "%s %s %f %f", entity, 'move', x, y )

while i uderstand the basic nature of the symbols i have no idea how to use them in such a way that i am looking for





i am looking for a way to try and get the following serialized

Code: Select all

payload = { player.x,player.y,req_type,data_index}


*send all that threw to the serverover the udp as a string if im not wrong
then re assemble it on the other side.

so how do i do this. you cant push a table threw can you ?



also here i also dont know whats going on much. i know they are un packing received data into variables but the symbols make no snese to me.

i marked it in the code lol

Code: Select all

data, msg = udp:receive()
 
      if data then -- you remember, right? that all values in lua evaluate as true, save nil and false?
 
         -- match is our freind here, its part of string.*, and data is
         -- (or should be!) a string. that funky set of characters bares some 
         -- explanation, though.
         -- (need summary of patterns, and link to section 5.4.1)
         ent, cmd, parms = data:match("^(%S*) (%S*) (.*)")   < ----------------------- here ( wtf??)
         if cmd == 'at' then
            -- more patterns, this time with sets, and more length selectors!
            local x, y = parms:match("^(%-?[%d.e]*) (%-?[%d.e]*)$") <---------------- yup ... drawing a blank 
            assert(x and y) -- validation is better, but asserts will serve.  < ----------- validation??
 
            -- don't forget, even if you matched a "number", the result is still a string!
            -- thankfully conversion is easy in lua.
            x, y = tonumber(x), tonumber(y)
            -- and finally we stash it away
            world[ent] = {x=x, y=y}
         else
            -- this case shouldn't trigger often, but its always a good idea
            -- to check (and log!) any unexpected messages and events.
            -- it can help you find bugs in your code...or people trying to hack the server.
            -- never forget, you can not trust the client! <---------- i get this
            print("unrecognised command:", cmd)
         end
 
      -- if data was nil, then msg will contain a short description of the
      -- problem (which are also error id...).
      -- the most common will be 'timeout', since we settimeout() to zero,
      -- anytime there isn't data *waiting* for us, it'll timeout.
      --
      -- but we should check to see if its a *different* error, and act accordingly.
      -- in this case we don't even try to save ourselves, we just error out.
      elseif msg ~= 'timeout' then 
         error("Network error: "..tostring(msg))
      end
   until not data 
 
end