Best way of serializing data for transmission over network interface

[steven777]

currently reading the lua 5.4.: http://www.lua.org/manual/5.1/manual.html#5.4.1


doesnt go into much detail about the patterns. or its over my head to an extent..

anyone got any like help for this.

[pgimeno]

That example uses a custom, very simple serialization scheme that can't serialize multiple strings properly. If you're interested in how some of these patterns work, take a look at this thread: https://www.love2d.org/forums/viewtopic.php?f=3&t=82551

If you want to serialize a flat table of data with a fixed number of values that do not include strings, you can use a separator of your choice that doesn't conflict with your data types (any symbol except '+' and '-' will probably do), e.g. '|'.

For numbers, I recommend this serialization method because it's mostly lossless: string.format("%.17g", value) (it's lossless if you don't have Infinity or NaN as a value).

For most other types, just tostring would do.

For example, if all these values are numbers, you can use:

Code: Select all

datastring = string.format("%.17g|%.17g|%.17g|%.17g", player.x, player.y, req_type, data_index}

This generates a string with the numbers separated by '|' e.g. '400|300|1|5'

To deserialize, you can use this:

Code: Select all

player.x, player.y, req_type, data_index = datastring:match('^(.-)|(.-)|(.-)|(.-)$')
-- check for nil here, if they are not nil then do this:
player.x = tonumber(player.x)
player.y = tonumber(player.y)
req_type = tonumber(req_type)
data_index = tonumber(data_index)

[steven777]

--------------------------
question 1
--------------------------


ok i have questions regarding your code at the moment.

^ start here in the string(beginning)
%. means all cearacters
17 ?? specifies the amount of cheraters
g doesnt seem to be on the list of classes? so did you mean d ?
() is the section..
| is the separator


now looking at the desalinizing code.. you have the following '^(.-)|(.-)|(.-)|(.-)$'

why are you using all characters when these are numbers ? ( . = all characters, - = 0 or more repititions)




--------------------
question 2
---------------------


also is there a way of making this accept strings of any length with all numbers. meaning i could take x,y,s,v,t,x (variabes that contain random numbers) and pass them threw at different times for certain )

in the desalinizing code above its a set amount of variables ^(.-)|(.-)|(.-)|(.-)$ exactly 4 variables. what if i wanted this to be a unknown set of numbers but like the first input is like a call function method so it knows how to handle the data no matter what strange string it gets. unless it really just doesn't know which gives and error.


----------------
question 3
-----------------


or what about sending a table inside of a table ?

here is n example of somthing i may use in a networking solution.

such as

reqtype = 1 being update or somthing. maybe like an in the game state. server would know how to handle the data up front id asume.

table = {req_type = 1, player_data = {--some data-- x,y,health}}

would i have to break it all down into one long string and have to re assemble it.

is this even possable ??

[pgimeno]

string.format does not work with patterns, it works with format strings. It's a bit confusing that both use % but they have nothing to do. From the manual:

• The format string follows the same rules as the printf family of standard C functions.

Googling for c printf format string: http://www.cplusplus.com/reference/cstdio/printf/

The format string can contain normal text and format specifiers starting with % as follows:

%[flags][width][.precision][length]specifier

The stuff in [ ] is optional. In the case of %.17g there are no flags, no width and no length. We're using .precision (.17) and a specifier of 'g' which means "the shortest of %e and %f". 'e' means "Scientific notation (mantissa/exponent), lowercase" and 'f' means "Decimal floating point, lowercase". So it picks up the shortest between decimal and scientific notation. For example, it will prefer "35.3" over "3.53e1" but also "1.2e20" over "120000000000000000000". The .17 means up to 17 decimal places. These are the minimum necessary to transform a double-precision number losslessly.

See also the examples in the link if you want to understand it better.

now looking at the desalinizing code.. you have the following '^(.-)|(.-)|(.-)|(.-)$'

why are you using all characters when these are numbers ? ( . = all characters, - = 0 or more repititions)

Because I'm only interested in the separators, really. Anything between the separators will be considered a number. If it's invalid, tonumber will convert it to 0 anyway. Just watch out that malicious clients may send NaN or Infinity, or invalid values. Check that the values are in a range you expect. Check explicitly for NaN. NaN is not equal to itself, so a way to check for NaN is: if var ~= var then --[[is NaN]] else --[[is not]]

also is there a way of making this accept strings of any length with all numbers. meaning i could take x,y,s,v,t,x (variabes that contain random numbers) and pass them threw at different times for certain )

in the desalinizing code above its a set amount of variables ^(.-)|(.-)|(.-)|(.-)$ exactly 4 variables. what if i wanted this to be a unknown set of numbers but like the first input is like a call function method so it knows how to handle the data no matter what strange string it gets. unless it really just doesn't know which gives and error.

...

or what about sending a table inside of a table ?

Then you're requiring generic serialization of a table. That's exactly what serializing libraries are designed for.

You can try to make your own if you wish. Good luck with that.

[Nixola]

If it's invalid, tonumber will convert it to 0 anyway.

[manual]tonumber[/manual] will return nil on invalid input.

[pgimeno]

[manual]tonumber[/manual] will return nil on invalid input.

Yes, my bad. Mixing languages, sorry

[steven777]

ok so now I have new questions here, you guys are really good . never had to f** with this but I'm glad you guys are here for people like me and I'm sure others have probably the same kind of questions.

my new question is how much data can be sent over the udp network layer. I am building on top of the udp transport layer and building my game from networking down. Centered around player interactions and such.


so lets say I am always going to send the same amount of data to the server (strictly just inputs) certain parts of the string need to be read first

here bellow is the mock packet assembly ( a table that holds all data a packet will ever contain being sent to a game server( also written in love2D)

Code: Select all

local packet_data = {

req_type = ([a number] command that are sent from the client to the server ( just words that are used in some if then statments on the server)
data_index = [a number] that tells the current message number being sent .
priority_index = ([a number]  not needed but it could be used to over ride a previous action from the same client[ probably not needed])

player.x                                       (value)
player.y                                       (value)
player. movment_direction            (value) 
player.angular_velocity                (value) 
player.angle                                (value) 
player.weapon                             (value) 
player.state                                 (value) 
player.ID =                                  (players id space
player.name =                             ( player name [ word or words... depends]
player. health =                           value
player.is_firing =                         true or false (word ig)
player.current_clip_size =           value


--- chat box stuff that also gets sent. if no message then the sever will notice from the index value and negate it 
sender_message = ( has limits and rules and all of that.
sender_indexMSG = 
sender_recall_index  =  
sender_to_player = 


}

so yea my question is it a good idea to make each packet uniform with its own look up values inside of it to like know what it is.

say the server gets the message and reads the first few bites of the string and says ok this is a player update and does what it needs to with the rest...
or should i say split packets up... one for messages and on for player stuff.




also whats the max amount of data that can be sent... is this to much ?
this is not my mmos player statistics but its close enough. I will maybe use 10 more variables or so.

[steven777]

so should i send all of the data like this or split it up and use string tags ??

[bjornbytes]

I wrote a small library called trickle (https://github.com/bjornbytes/trickle) for this purpose. You might get some ideas from there

[steven777]

right now i'm sending strings that sever knows if it doesn't fit the parameters it drops it as invalid and that user may be cheating so it flags it.

right now i have 30 users connected to a server running on my desktop. it runs over udp but i have tcp elements codded on top of it. I coded a master server and some shard servers/instance servers in the past few days iv been quite at work.


but I wondering if I am still vulnerable to a hack of some type..

i send put a string of data to the server, the server reads the first few bits of the string and determine what parameters are to be called. if a param isnt there it does not freeze my server atm or cause any harm it just simply says that that the string is not valid and doesn't preform operations.


its def not a lib of any sorts. its simply a server smashed together to emulate connection atm. i am starting to send real data over it now. like account information etc.