Asynchronous resource loading

[leiradel]

Hi All,

I was writing a 2D game engine in C++ when I knew about Löve. My engine, which is almost completed, has these characteristics:

. Uses SDL to initialize the graphics mode, load images, load and play sounds and music, load TTF fonts and render text with them.
. Uses a resource package system which can read from ZIP files or folders.
. Uses Lua as the main programming language for the game.
. Can load resources asynchronously.
. Can upload images as textures to the GPU and render text asynchronously.
. Can dynamic load external libraries which extend the engine's functionality (was WIP).

When I saw Löve for the first time I thought it was exactly what I wanted. After playing with it a bit and reading the docs I saw that it was close, but not quite there. The thing I miss the most is asynchronous functionalities of my engine, which could for example load everything needed to show the main game menu while the intro screen was being shown.

I'm not going back to my engine since Löve is much more mature, already tested with different platforms, has Box2D already integrated and a great community around it, and has a great license. But I do miss the asynchronous things.

I see there is something going on with threads in the repository. Will it make it possible to load things asynchronously? I wanted to take a look at the source code but I'm failing to see where I can download it. The Mercurial repository doesn't offer a .tar.gz download or similar, does it?

Cheers,

Andre

[bmelts]

Hi, Andre!

To answer your first question: yes, currently in the repository there is a love.thread module which will enable you to load (and do other things) asynchronously. If you prefer to use currently available (and tested) technology, you can also see if you can bend Lua coroutines to your whim; they're not quite threads, but it is possible to write a loader with them where it loads a single resource before yielding back to the main LÖVE loop so it can update the screen, which then yields back to the coroutine so it can load the next resource. Lather, rinse, repeat. This is a lot nicer than just loading every resource in one go before doing anything else, but it's not truly asynchronous.

If you just want to view the source code, you can browse through the source tree on the repository page; click the "files" link at the top, and navigate your way through the hierarchy. The thread code is in src/modules/thread/sdl.

If you'd like to actually try out the experimental thread support, you'll need to install Mercurial in order to download the bleeding-edge source for LÖVE - SourceForge doesn't, as far as I know, offer a "download files" button. Once downloaded, just run hg clone http://love.hg.sourceforge.net:8000/hgroot/love/love in whatever directory you'd like to put the love folder in. From there, build as appropriate for your system; it comes with Visual Studio and Xcode projects for Windows/Mac, or if you're building on Linux, just cd to the root of the love folder and run sh platform/unix/automagic. However you choose, you'll eventually end up with an executable of the bleeding-edge of LÖVE, and can mess around with love.thread to see if it fits the bill.

Feedback on whether it can do what you need it to would be valuable as well, if you feel like giving it.

I hope you enjoy using LÖVE!

[leiradel]

To answer your first question: yes, currently in the repository there is a love.thread module which will enable you to load (and do other things) asynchronously.

Great, I'll try to grab the source code and compile it to test if it can do what I want.

One more question about this: how and when images are transformed into textures and sent to VRAM? Can I make it happen asynchronously, like uploading it right after loading it? I remember during my research that texture uploading could no be made async in OpenGL, so I ended up uploading textures in small chunks in the game loop. It worked very well, big textures would be completed uploaded very fast and without freezing the game loop.

If you prefer to use currently available (and tested) technology, you can also see if you can bend Lua coroutines to your whim;

Yeah, I used coroutines to implement an app which would have to visit hundreds of HTTP servers and note their response codes. It would probably work for small resources but I prefer to use threads. Maybe the texture uploading could be done using coroutines?

However you choose, you'll eventually end up with an executable of the bleeding-edge of LÖVE, and can mess around with love.thread to see if it fits the bill.

LÖVEly!

Feedback on whether it can do what you need it to would be valuable as well, if you feel like giving it.

Sure.

Just one more question: I think I've read in the Lua mailing list by the time threads were added to the VM that when Lua is compiled with threads support each access to the global table would be guarded with a mutex and thus would have some overhead. Is it what love.thread does? If it is maybe conditional compilation of that module would be cool. And I'd then move away from it in favor of coroutines.

Thanks,

Andre

[bmelts]

An image is converted to a texture when it's created; from the Lua end, that would be when you call love.graphics.newImage. The data from the given file is loaded into memory, OpenGL generates a texture, sets some relevant parameters, and passes glTexImage2D the aforementioned data. After that, it's all in OpenGL's hands.

If you want to have the data already in memory, but don't want to turn it into a texture immediately, you could do something like this:

Code: Select all

myImageData = love.image.newImageData("myImage.png")
-- hold on to myImageData until you want to give it to OpenGL to deal with, at which point:
myImage = love.graphics.newImage(myImageData) -- texturing ahoy

As for love.thread - I'm afraid it's a bit beyond me. bartbes is the one who implemented love.thread, so ask him any questions about overhead or other details. That said, you can always disable it in conf.lua if you don't want threading support and any issues it might bring with it.

[bartbes]

Indeed you can't create images and such in any other thread than the main thread, but as far as I can tell sounds do work.
And there are no mutexes used for lua, though it seems the GC is a bit unstable (the GC in love), so we might need to add some mutexes there.

[leiradel]

Indeed you can't create images and such in any other thread than the main thread, but as far as I can tell sounds do work.

So we're back to coroutines. It's not a big issue unless there's a huge image to be loaded, but then the programmer can cut the images in smaller pieces and use them to blit the entire image on the screen.

And there are no mutexes used for lua, though it seems the GC is a bit unstable (the GC in love), so we might need to add some mutexes there.

Hum... I think that to properly implement threads in Lua you have to define some macros, LUA_LOCK and LUA_UNLOCK if memory doesn't fail, and then compile it. So even if love.threads can be disabled in the conf file the overhead of locking and unlocking the mutex will be there because it's compiled in the VM.

[bartbes]

No, it isn't necessary, threads have different states. (and a thread can only access its own state)

[leiradel]

How are you guys planning to allow data exchange between threads?

[Robin]

How are you guys planning to allow data exchange between threads?

Two words: Magic. Ponies.

[leiradel]

How are you guys planning to allow data exchange between threads?

Two words: Magic. Ponies.

Hahaha. What are threads good for in LÖVE then?