[SOLVED] Love2D & Lua with a local database?

BOT-Brad · Post by **BOT-Brad** » Sat Jan 24, 2015 10:34 pm

Hi folks,

Just a quick question. I'm working on a project where I need to pull in and process a fair whack of data. This data is simply a locally stored file with a lot of config and setup data for some geographical mapping and plotting systems. While testing and developing the project, I have been storing a small amount of the data as native lua table objects, but maintaining the large amount of data this way is virtually impossible. It was no issue when there were only 50 or so entries, but the full database has over 40,000 entries and it becomes a pain to manage such a large lua table easily.

Basically, is there any lightweight and platform-independent way to read some form of local database (Be it SQLite or similar) in Love/Lua without much hassle? Alternatively if there isn't, then is there a better way to manage a 40,000 line .txt file to easily manage the data without major headaches?

Cheers.

Positive07 · Post by **Positive07** » Sun Jan 25, 2015 1:49 am

Some questions:

Is the data in a database like SQL or just in a txt file?

Do you need all the data at the same time?

The data has a predefined format? say JSON or XML

Have you used LÖVE in some other project?

Isnt there a tool that does the same job already?

Have you considered other alternatives instead of LÖVE?

Do you have experience programming?

Is this project something you MUST do or something you wanna do?

ivan · Post by **ivan** » Sun Jan 25, 2015 5:45 am

Hey. 40k entries is not a lot in terms of memory.
So the question becomes, how do you index/look up entries.
With games the general approach is to index the entries at load time so you can do quick lookups later.

Also, I assume you have a sensible idea of parallel arrays so:
nested:

Code: Select all

list[1] = { x = 1, y  = 1 },
list[2] = { x = 5, y = 10 },
...

parallel:

Code: Select all

list_x[1] = 1
list_y[1] = 1
list_x[2] = 5
list_y[2] = 10
...

For large lists, the bottom approach takes up a lot less memory.

Do you need all the data at the same time?

This is an important question too.

BOT-Brad · Post by **BOT-Brad** » Sun Jan 25, 2015 3:24 pm

Is the data in a database like SQL or just in a txt file?
It is currently in a MySQL database, but I can retrieve it in various other formats such as a local SQLite database, JSON or XML if needed.

Do you need all the data at the same time?
I only need to read and process the data at a single point in time, I do not need to store it anywhere.

The data has a predefined format? say JSON or XML
As above, I can export to JSON or XML but I was wondering if I could load directly from the database instead.

Have you used LÖVE in some other project?
I'm kind of new (ish) to Love and Lua in general.

Isnt there a tool that does the same job already?
I found a few, but they all are platform dependent and a bit overkill for my needs.

Have you considered other alternatives instead of LÖVE?
It would be trivial for me to do this in Java or Python, but I found Love to be a really quick prototyping environment.

Do you have experience programming?
Yes, I have worked in software development for 4 years and consider myself a decent programmer.

Is this project something you MUST do or something you wanna do?
This was purely a learning project and nothing work-related.

Hey. 40k entries is not a lot in terms of memory.
So the question becomes, how do you index/look up entries.
With games the general approach is to index the entries at load time so you can do quick lookups later.
I'm not really worried about mem usage, as although there are 40,000~ entries in the database, the data isn't just loaded in, it gets read and processed into a different format. The resulting lua table is then only around 6,000 indexes long, rather than 40,000.

Positive07 · Post by **Positive07** » Sun Jan 25, 2015 3:44 pm

Okey then you dont need to translate, JSON is really easily interpreted by lua already and there are libraries that can help you read it (For example this one which makes use of LPeg, I have a project that uses LPeg in LÖVE so I can help you on setting that up If you wanted)

If you dont need to compare ALL entries or to have them all in memory, you could load a chunk, do the processing on that chunk, store it, read another chunk and so on...

If you could go with that approach then you could even use threads in your app which would mean you could do it faster

If you wanted to, you could use an FFI binding to SQL. I recomend this over a C library since you wouldnt need to recompile LÖVE, I have never worked with any of this binding, so I dont know if they are cross platform or not, still, you should check them out

kikito · Post by **kikito** » Sun Jan 25, 2015 8:18 pm

In your case, I would export the data to a .lua file, require it, process it, and then export the 6000 resulting records in a format which is useful to you. It is alwo worth noting that you can probably plenty of other tools than LÖVE for that processing.

BOT-Brad · Post by **BOT-Brad** » Sun Jan 25, 2015 8:26 pm

kikito wrote:In your case, I would export the data to a .lua file, require it, process it, and then export the 6000 resulting records in a format which is useful to you. It is alwo worth noting that you can probably plenty of other tools than LÖVE for that processing.

I'm using LOVE because once all the data is processed, the actual 'application' starts and shows a nice interactive world map with the location data marked down. I'm not entirely sure how to go about converting the database into a .lua file though...

s-ol · Post by **s-ol** » Sun Jan 25, 2015 9:21 pm

BOT-Brad wrote:
kikito wrote:In your case, I would export the data to a .lua file, require it, process it, and then export the 6000 resulting records in a format which is useful to you. It is alwo worth noting that you can probably plenty of other tools than LÖVE for that processing.
I'm using LOVE because once all the data is processed, the actual 'application' starts and shows a nice interactive world map with the location data marked down. I'm not entirely sure how to go about converting the database into a .lua file though...

export as json, regex replace; write extremely small tool that does, etc.

BOT-Brad · Post by **BOT-Brad** » Sun Jan 25, 2015 11:20 pm

S0lll0s wrote:
BOT-Brad wrote:
kikito wrote:In your case, I would export the data to a .lua file, require it, process it, and then export the 6000 resulting records in a format which is useful to you. It is alwo worth noting that you can probably plenty of other tools than LÖVE for that processing.
I'm using LOVE because once all the data is processed, the actual 'application' starts and shows a nice interactive world map with the location data marked down. I'm not entirely sure how to go about converting the database into a .lua file though...
export as json, regex replace; write extremely small tool that does, etc.

Yep, I decided to do this and just cycle through the json and converting the data into a lua table and it works great.
Still, are there any good platform-independent integration libs for Lua when working with SQL db's? Would be interesting to know.