Using cameras

[leiradel]

Hi All,

I've tried to support cameras ion LÖVE some time ago, using an ImageData that would change its contents as new frames arrived from the camera. It was a bad way to do it, and the changes never see the light of a pull request.

Now that LÖVE has a video module, I wonder how cameras could be implemented? I think that the module is tied to Theora videos, and I'm not sure how to add another stream that grabs frames from the camera.

Any thoughts?

Cheers,

Andre

[Tjakka5]

Im guessing it would work similair to love-microphone:
https://github.com/LPGhatguy/love-microphone

[leiradel]

Interesting in that it doesn't have a single line of native code. In my case I need to compile a camera library which is written in C, so I think I'll just do the Lua bindings myself.

[pgimeno]

Look up FFI, it's for creating bindings to existing C libraries. Edit: and that's what love-microphone uses.

[leiradel]

Yes, I saw it being used in the source code. The thing is, the library is, AFAIK, not available by default in any OS, so it have to be compiled for the module to work, not counting the code to interface with LÖVE's video module.

Since there will be a compilation step, I'll just write the binding code myself and avoid the dependency on FFI.

My question was more about how to implement a VideoStream, but slime and bartbes already helped me in that regard at the #love IRC channel.

[Positive07]

FFI comes with LuaJIT which LOVE uses so you can require "ffi" and assume it will work anywhere.

The only compilation you would need to do is that of the library that access the camera.

You can then use FFI to call functions from the camera C library and interface them with LOVE functions. Then provide a Lua API for the user of the library.

This is what projects like love-microphone and LOVE3D do, but they require C libraries that are already part of LOVE like OpenAL... A better example may be luajit-request

[leiradel]

I'm not sure what the benefit would be. I also don't know how to extend the VideoStream class and implement its abstract methods. I really don't see the benefit, specially given that the binding is already coded.

[Positive07]

If the binding is already coded then you should probably use that, FFI is just another way of binding which requires no C code nor compiling.

Even then that binding won't work with LÖVE since there are no Lua API's to work with [wiki]VideoStream[/wiki]s objects (and with good reason, since you can't probably generate new frames of data every frame from Lua). This means that if the binding you have exposes a Lua API then you can't use that, you need to work on the C side. This would probably also hold true even if using FFI, so keep on using C, those bindings will help as a reference and you will reuse part of the code, I'm pretty sure.

So in order to create a new [wiki]Video[/wiki] object you will need a VideoStream, since LÖVE will only create VideoStreams from Ogg Theora files you won't be able to use any of LÖVE codes (unless you modify them which I don't recommend) to create your VideoStream object. Instead you will create an userdata that mimicks the VideoStream object ENTIRELY! even if you can't rewind the feed from the camera and such, you will need to implement dummy functions. You won't be able to add functionality to the Video object, except through your library. All this limitations come from trying to use the Video object LÖVE provides. If you are still on this ship then let me go on

First you will need to create a VideoStream userdata which you will return. Note that you need to use the same ID defined in this enum, and your object needs to provide the same struct so that when LÖVE calls the methods they are the same.

Then the userdata is sent to [wiki]love.graphics.newVideo[/wiki] which is defined in line #916 to #930 of wrap_Graphics.cpp here it checks if the userdata is a VideoStream and creates a new Video instance.

Then comes the fun part, this Video object needs to be rendered to the screen, in order to do this love calls Video draw function which creates a new Mesh and loads a Texture to it, this texture is the buffer you need to fill in your thread with the actual data from the camera, and then the Video get's it through the VideoStream -> getFrontBuffer() used in lines #80 and line #155 of Video.cpp, this function should return the frontBuffer which you should load on a different thread so that you don't lock program execution, this is done in threadedFillBackBuffer() which fills a second buffer and then the buffers are swapped. Note that the frontBuffer needs to have the same struct as that of love so all the fields are there... Again I don't know how you can acomplish that.

Also don't forget to implement all the other methods VideoStream and Video provides or you may crash everything.

Oh and I haven't even started about Audio... Video is all over the place. Have fun and enjoy your coding nightmares.

PS: This module is not encapsulted at all, and right now is only designed to play Theora Ogg videos... We can ask the developers to make this better but don't expect much though. I would love to see a C API to implement something in the C side that can behave as VideoStreams or something like that.

[leiradel]

The guys on #love gave me some directions, but you wrote some pretty detailed instructions so thanks!

I've implemented cameras in LÖVE before, I've changed the ImageData class to add a property that would tell the graphics module that the data was dirty and needed to be uploaded to the GPU again. The the camera thread could just fill in the pixels and set the dirty flag to true.

It worked somewhat ok, but I was having some hiccups on the frame time, and LÖVE was crashing upon exit. Now that there's a video module, I hope things will go smoother.

Cheers,

Andre

[Positive07]

Well if you could completely mimick the VideoStream, you would get EXTREME performance boost, since it would be like having two ImageDatas that you can fill in a different thread, and then just swapping between the most recent one and the old one... This is basically what the Video module does, also it performs a sync with the audio and the main thread so that all three share their FPS and don't get out of sync. The module is great, but generating compatible userdata will be hard, since you have to provide the same structure, which I have no idea how you would do without modifying LÖVE itself.

If you are modifying LÖVE then copy the entire Video directory, and rename/replace all the Video names with Camera or something like that. Don't change the VIDEO_VIDEO_STREAM_ID though, since that means that the object is a VideoStream object. Then replace whatever is completely Theora related with the camera code (basically filling those buffers and whatever happens with the sound) and you are done, as I said you may need to create some dummy functions for rewind, seek and tell but that is little to no problem