Playing Video. mjpeg decoder in pure lua. With audio!
Playing Video. mjpeg decoder in pure lua. With audio!
Ok, ok. The subject is a joke. A little bit.
But, anyway, this can be considered as mjpeg decoder. We decode jpeg's? Yes. We get motion? Yes! Is it pure Lua code? No. But only thing it needs - our lovely framework.
When I was hit by this idea, i didn't know what something similar was already done.
My variant has smaller .love file size. Smaller memory usage (350mb peak for hq version), but very low quality.
So, anyway, try out this stuff! Any kind of feedback are welcome!
But, anyway, this can be considered as mjpeg decoder. We decode jpeg's? Yes. We get motion? Yes! Is it pure Lua code? No. But only thing it needs - our lovely framework.
When I was hit by this idea, i didn't know what something similar was already done.
My variant has smaller .love file size. Smaller memory usage (350mb peak for hq version), but very low quality.
So, anyway, try out this stuff! Any kind of feedback are welcome!
- Attachments
-
- mjpeg-player_hq.love
- This one needs more CPU power
- (6.1 MiB) Downloaded 641 times
-
- mjpeg-player.love
- (1.48 MiB) Downloaded 573 times
- josefnpat
- Inner party member
- Posts: 955
- Joined: Wed Oct 05, 2011 1:36 am
- Location: your basement
- Contact:
Re: Playing Video. mjpeg decoder in pure lua. With audio!
This is super cool! I knew it was only a matter of time before someone came in and did video better than my basic attempt. I have questions!
What kind of performance increase did you get from putting 16 frames in one image?
How big was the hq file before hand ?
Can you share the scripts you used to generate the .ogg file, and the .jpgs?
What kind of performance increase did you get from putting 16 frames in one image?
How big was the hq file before hand ?
Can you share the scripts you used to generate the .ogg file, and the .jpgs?
Missing Sentinel Software | Twitter
FORCIBLY IGNORED.
<leafo> when in doubt delete all of your code
<bartbes> git rm -r *
<bartbes> git commit -m "Fixed all bugs"
<bartbes> git push
FORCIBLY IGNORED.
<leafo> when in doubt delete all of your code
<bartbes> git rm -r *
<bartbes> git commit -m "Fixed all bugs"
<bartbes> git push
Re: Playing Video. mjpeg decoder in pure lua. With audio!
I get source video from Wikipedia.
Then converted it to image sequence (and also scaled to needed width*height) with ffmpeg. Then images was sticked with montage.
I will post my scripts from home. You should wait something about 12 hours.
I do not know about any performance boost with 16 frames per image. The idea was: Load one second of animation and while it is displayed load another second.
In LQ example this worked great. In HQ i get bad performance on my 1GHz Atom processor.
Now i have some ideas, how this can be improved. I will try them at home and post results.
Then converted it to image sequence (and also scaled to needed width*height) with ffmpeg. Then images was sticked with montage.
I will post my scripts from home. You should wait something about 12 hours.
I do not know about any performance boost with 16 frames per image. The idea was: Load one second of animation and while it is displayed load another second.
In LQ example this worked great. In HQ i get bad performance on my 1GHz Atom processor.
Now i have some ideas, how this can be improved. I will try them at home and post results.
- retrotails
- Party member
- Posts: 212
- Joined: Wed Apr 18, 2012 12:37 am
Re: Playing Video. mjpeg decoder in pure lua. With audio!
This is awesome. I assume it would work with lossless images as well?
- Roland_Yonaba
- Inner party member
- Posts: 1563
- Joined: Tue Jun 21, 2011 6:08 pm
- Location: Ouagadougou (Burkina Faso)
- Contact:
Re: Playing Video. mjpeg decoder in pure lua. With audio!
I won't comment the code, as I didn't took a look at it - yet -, but I was really impressed.
Amazing work!
Amazing work!
Re: Playing Video. mjpeg decoder in pure lua. With audio!
Code is very ugly.
I just made some tests with different jpeg images. Looks like it is faster to load many small images than loading one big. 320x200 image loads 10 times faster than 640x400. I think image size has some limit after which it slows down a lot.
I just made some tests with different jpeg images. Looks like it is faster to load many small images than loading one big. 320x200 image loads 10 times faster than 640x400. I think image size has some limit after which it slows down a lot.
Re: Playing Video. mjpeg decoder in pure lua. With audio!
Okay, i wash my hands. I get all fun from this idea. Maybe I come back to it someday. So now it's your turn - do what you want, but do not forget to share your beautiful results
With good fine-tuning this method can allow you to have a nice cutscene in your game, played at full 25fps with resolution around 800x480. Most important here - size of images with video frames. Looks like what they should be not bigger than 800x800. So, just experiment. Maybe your choice will be one frame per image, or two.
Also remember what you can use anamorphic widescreen image.
Take a look at mjpeg_player_2hq.love demo. I think it is very cool to have such video just in 8mb of data.
Another idea what can be added to this code - "adaptive bitrate". Your code can switch to lowres version if there is not enough CPU power. Add 3mb of data and you have three versions of your cutscene - HQ, normal and low. Adding automatic switch of quality depending on available CPU should be an easy task.
Or, you can make your video very low quality and low res (180x100 for example) and enhance it with some artistic shader. Take a look at mjpeg_player_shader.love. Only 1.5mb of data, but with specially created animation it can be very cool and be a part of atmosphere of your game. 7 minutes of cutscenes just in 10mb! I think its nice option.
About the idea itself:
In simple words this is just a big SpriteBatch what streams it's frames from disk (frames are loaded just before they get to the screen and almost instantly get lost).
Here are not much difference with josefnpat code. I just used jpg instead of png (this saves a lot of space) and added helper thread what loads images for me (so there is no locking, one thread loads frames, another shows animation).
Now i describe process of video preparation:
You wil need folloeing tools: Python, ffmpeg and ImageMagik. this links are for WinUsers. If you with Linux, you should know what to do. (Sorty thous ones from the Mac Side, i do not know how to help you with this. Besides, you has cookies)
Now, with installed utils perform following steps:
1) Convert video to standalone frames:
2) Combine individual frames (if you need this) With following Python script which uses montage utility from ImageMagik:
3) Extract audio from source file:
This is for case when source audio already in Ogg Vorbis format. If this is not true, google for proper solution.
4) Now put all this in your Love project folder. take a look at my examples, its pretty clear how this should be done.
Then, modify settings. In my main.lua they looks this way:
Now, you can play your video.
Happy hacking and don't forget to have fun!
With good fine-tuning this method can allow you to have a nice cutscene in your game, played at full 25fps with resolution around 800x480. Most important here - size of images with video frames. Looks like what they should be not bigger than 800x800. So, just experiment. Maybe your choice will be one frame per image, or two.
Also remember what you can use anamorphic widescreen image.
Take a look at mjpeg_player_2hq.love demo. I think it is very cool to have such video just in 8mb of data.
Another idea what can be added to this code - "adaptive bitrate". Your code can switch to lowres version if there is not enough CPU power. Add 3mb of data and you have three versions of your cutscene - HQ, normal and low. Adding automatic switch of quality depending on available CPU should be an easy task.
Or, you can make your video very low quality and low res (180x100 for example) and enhance it with some artistic shader. Take a look at mjpeg_player_shader.love. Only 1.5mb of data, but with specially created animation it can be very cool and be a part of atmosphere of your game. 7 minutes of cutscenes just in 10mb! I think its nice option.
About the idea itself:
In simple words this is just a big SpriteBatch what streams it's frames from disk (frames are loaded just before they get to the screen and almost instantly get lost).
Here are not much difference with josefnpat code. I just used jpg instead of png (this saves a lot of space) and added helper thread what loads images for me (so there is no locking, one thread loads frames, another shows animation).
Now i describe process of video preparation:
You wil need folloeing tools: Python, ffmpeg and ImageMagik. this links are for WinUsers. If you with Linux, you should know what to do. (Sorty thous ones from the Mac Side, i do not know how to help you with this. Besides, you has cookies)
Now, with installed utils perform following steps:
1) Convert video to standalone frames:
Code: Select all
ffmpeg.exe -i Big_Buck_Bunny_Trailer_400p.ogg -s 640x480 -f image2 ffmpeg_temp/%05d.png
Code: Select all
#!/usr/bin/python2
import os
import math
total_frames = 813
cols = 1 # number of frames in each row
rows = 2 # number of frames in each column
frame_num = cols * rows
cmd_prefix = "montage -format jpg -quality 35 -geometry +0+0 -tile"
# Use this variant, if you want to restrict montage with only two frames in a row. do you see -tile parameter?
# cmd_prefix = "montage -format jpg -quality 35 -geometry +0+0 -tile 2x "
for j in range(0,1 + int(math.ceil(total_frames / frame_num))):
cmd = cmd_prefix
for i in range(1, frame_num + 1):
k=j * frame_num + i
if k > total_frames:
k = total_frames
cmd = cmd + str(k).zfill(5) + ".png "
cmd = cmd + "v_" + str(j + 1).zfill(5) + ".jpg"
os.system(cmd)
print (cmd)
Code: Select all
ffmpeg.exe -i big_buck_bunny_480p_stereo.ogg -vn -acodec copy audio.ogg
4) Now put all this in your Love project folder. take a look at my examples, its pretty clear how this should be done.
Then, modify settings. In my main.lua they looks this way:
Code: Select all
local frame_width, frame_height = 180, 100 -- this is size of individual frame
local scale_factor = 4 -- scaling factor. In this case, resulting video will be 720x400
local frames_in_row, rows_num = 5, 5 -- How many frames in your images?
local video_fps, video_spf = 25, 1/25 -- Frames per second and seconds per frame
Happy hacking and don't forget to have fun!
- Attachments
-
- mjpeg_player_shader.love
- (1.48 MiB) Downloaded 605 times
-
- mjpeg_player_2hq.love
- (7.98 MiB) Downloaded 345 times
Re: Playing Video. mjpeg decoder in pure lua. With audio!
Awesome!
Put this in loader_thread.lua:
and this in main.lua
Then the memory usage gets much lower -- 25mb
Put this in loader_thread.lua:
Code: Select all
while running do
collectgarbage("collect")
Code: Select all
function love.update(dt)
collectgarbage("collect")
- dreadkillz
- Party member
- Posts: 223
- Joined: Sun Mar 04, 2012 2:04 pm
- Location: USA
Re: Playing Video. mjpeg decoder in pure lua. With audio!
This is pretty amazing. Thanks for the code!
- Roland_Yonaba
- Inner party member
- Posts: 1563
- Joined: Tue Jun 21, 2011 6:08 pm
- Location: Ouagadougou (Burkina Faso)
- Contact:
Re: Playing Video. mjpeg decoder in pure lua. With audio!
Yes, that may work. Because, each loop update, the previously loaded userdata are no longer accessible, and thus can be tagged as garbage, and be cleaned fro memory.NÖÖB wrote:Awesome!
Put this in loader_thread.lua:and this in main.luaCode: Select all
while running do collectgarbage("collect")
Then the memory usage gets much lower -- 25mbCode: Select all
function love.update(dt) collectgarbage("collect")
But, what if we need some full video playing features ? go forth, go back , stop, resume, etc...
Which kinda make me think that the source can be optimized, for to enhance speed, or lower memory use.
@flashkot : For what reason did you used love.thread, by the way ?
Plus, did you envision some sort of streaming ? I mean, load some frames, display them, and progressively load the next frames in background while playing the video. It may reduce the reduce the memory usage, but i don't know if that would be faster. That's jus some random ideas I am throwing on the table, so that we can all learn from that.
Thanks for the amazing work, though.
Who is online
Users browsing this forum: No registered users and 12 guests