zlib/gzip in lua?

[Kadoba]

Hey, I'm working on a Tiled map loader for love. I know there are a couple of other loaders out there but they're pretty basic and I'm aiming for full features/compatibility. Tiled can encode the map data into 5 formats: comma seperated value (CSV), base64, gzip, zlib, and XML. Currently I have all but gzip/zlib working and I'm having some trouble and was hoping the more technical-savvy people here could help me.

I have found http://lua-users.org/wiki/ModuleCompressDeflateLua which looks promising but it's very poorly documented (as in not at all). I've dug through the uncommented code for hours and I haven't really made any progress. It looks like you feed the gunzip function in deflatelua.lua a table with the fields "input" and "output". The input can be a string but the output has to be either a file or a function which handles bytes individually. I wrote a simple function that prints all bytes as characters but when i try and feed a gzip encoded string and the function to deflate.gunzip nothing happens. I simply get a cryptic "(no error message)"

Here is a sample gzip string:
H4sIAAAAAAAACw3DhwnAMAwAMP8P2Rdk9s1KoBQR2WK12R1Ol9vj9fn5A/luZ4Y4AAAA
which should be decoded to a base64 string:
jQAAAI4AAACPAAAAkAAAAJEAAACSAAAAkwAAAKEAAACiAAAAowAAAKQAAAClAAAApgAAAKcAAAA=

I'd really like to get the decompression working as everything else is nearly finished but this is above me. A little advice or guidance would be great.

[Kadoba]

Okay I made some progress and i think I figured out what I was doing wrong. It's not completely working yet but I think I'm heading in the right direction.

[tentus]

Once you have something be sure to post it, I'm sure others would be interested in the working code.

[Kadoba]

Once you have something be sure to post it, I'm sure others would be interested in the working code.

Will do. In the mean time I'm stuck again. I'll explain my situation so far.

First of all I was mistaken when I said that the gunzipped string in my original post should be decoded to a base64 string. That's where I went wrong. In reality the gunzipped string was actually in base-64 itself. So I had to decode the data from base-64 into bytes and then feed those bytes as a regular string into the gunzip function found in deflatelua.lua. Thankfully I've already written a base64 decoder so that wasn't so bad. So base64 isn't an issue any more.

Okay so here's the gunzip string again:
H4sIAAAAAAAACw3DhwnAMAwAMP8P2Rdk9s1KoBQR2WK12R1Ol9vj9fn5A/luZ4Y4AAAA

This is suppose to be converted into 32-bit unsigned integers:
141,142,143,144,145,146,147,161,162,163,164,165,166,167

What got me excited is that I actually got the gunzip function to spit out the correct sequence of bytes for this:
141, 0, 0, 0, 142, 0, 0, 0, 143, 0, 0, 0, 144, 0, 0, 0, 145, 0, 0, 0, 146, 0, 0, 0, 147, 0, 0, 0, 161, 0, 0, 0, 162, 0, 0, 0, 163, 0, 0, 0, 164, 0, 0, 0, 165, 0, 0, 0, 167, 0, 0, 0

The problem is that it keeps going after this and proceeds to spit out these weird bytes as well:
166,0,0,0,0,0,0,0,145,143,161,0,145,143,161,163,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

And then after that I get and unexpected end of file error . I thought this was something I could work with and it was probably an error in my base64 decoder but after testing it some more I realize this isn't the case. And the thing is depending on the data I feed it, It will either produce perfect results or it won't spit out anything before giving me an end of file error.

I've attached my base64 decoder if anyone else wants to give it a go. To use it just call "decode(mode, base64string)" where mode is either "int" "byte" or "string". Int will produce a table of 32-bit numbers, byte will produce a table of 8-bit numbers, and string will just be a string of 8-bit characters.

Here's an example on how to use the gzip script I linked in my opening post:

Code: Select all

deflate = require("deflatelua.lua")
base64 = require("base64.lua")
t = {}
t.input = base64.decode("string", "H4sIAAAAAAAACw3DhwnAMAwAMP8P2Rdk9s1KoBQR2WK12R1Ol9vj9fn5A/luZ4Y4AAAA")
t.output = function(byte) print(string.char(byte)) end
deflate.gunzip(t)

[Kadoba]

Hey, good news guys and gals. I found a sneaky bug in my base64 decoder that was causing most of the problems. I got gzip functionality working soon after. I contacted the author of the deflate script and he was kind enough to implement support for zlib headers after I asked about them. So I currently have all formats working for my Tiled loader and am very close to being done.

I've uploaded the fixed base64 script if anyone wants to use it. You can parse gzip strings in the same manner as in the example in my third post. zlib is done in the same way except instead of using deflate.gunzip() you use deflate.inflate_zlib().

[kikito]

Hey, good news guys and gals. I found a sneaky bug in my base64 decoder that was causing most of the problems. I got gzip functionality working soon after. I contacted the author of the deflate script and he was kind enough to implement support for zlib headers after I asked about them. So I currently have all formats working for my Tiled loader and am very close to being done.

I've uploaded the fixed base64 script if anyone wants to use it. You can parse gzip strings in the same manner as in the example in my third post. zlib is done in the same way except instead of using deflate.gunzip() you use deflate.inflate_zlib().

Thanks for the efforts on this. I'm interested in your zip lib, but I'm even more interested in your tiled-loading lib

[vrld]

For base64 encoding and decoding, you could also use the mime module of luasocket, like so:

Code: Select all

local mime = require 'mime'
encoded = mime.b64('hello, world') --> aGVsbG8sIHdvcmxk
decoded = mime.unb64(encoded) --> hello, world

That way you have one less component to worry about.

And like kikito, I am very interested in your tile-loading lib

[bartbes]

And love itself has a b64 decoder as well (in FileData).

[Kadoba]

Thanks for the efforts on this. I'm interested in your zip lib, but I'm even more interested in your tiled-loading lib

I only created the base64 decoder. A really hokey one. The gzip stuff was done by this fine dude: https://github.com/davidm/lua-compress-deflatelua

For base64 encoding and decoding, you could also use the mime module of luasocket, like so:

Yeah that would have saved me a lot of time. Mine's working now and I learned a lot so it's not too big of a deal.

And love itself has a b64 decoder as well (in FileData).

I saw that but I couldn't figure out how to get it to work with strings.


*edit*
Hooray! I've finished the first release of the Tiled loader. You can find it here: http://love2d.org/forums/viewtopic.php?f=5&t=2567