[Proposal] Better multimedia formats.

[Imagic]

Hi,

To cover a lot of use cases, a core set of multimedia formats could be added to LÖVE.

The following list is what I would currently want to implement for a game engine:
- video => webm + vp8 + opus
- audio (music/sound) => ogg + opus (lossy) | ogg + flac (lossless)
- audio (VoIP, packet-based) => opus
- image => webp

All of these are royalty-free formats with implementations under permissive licences like BSD.

Adding formats is difficult, so I think the most interesting ones to add to LÖVE would be WebP and Opus (there is already WAV for lossless audio or Ogg/Theora instead of WebM/VP8; which can use Opus for audio).

WebP:
- simple API (libwebp)
- lossy/lossless with alpha format; seems to perform better (quality/space) than PNG and JPEG
- sequences/animations (could be loaded as array textures)
- JPEG has been removed from LÖVE, WebP can fill the gap

Opus:
- one of the best lossy formats for music/voice/sound
- also designed for VoIP (packets)

WebP is supported by Chrome/Chromium and recently by Firefox, here are examples of a WebP animation as an atlas and a sequence, with a black or alpha background (125 frames, 128x128).

2,0M anim_add_animated.webp (forgot to make this one loops)

888K anim_add_animated_lossy.webp (q = 100 to reduce artifacts)

1,5M anim_alpha_animated.webp

696K anim_alpha_animated_lossy.webp (q = 75)


atlas:
1,9M anim_add.webp
336K anim_add_lossy.webp (q = 75)
1,3M anim_alpha.webp
652K anim_alpha_lossy.webp (q = 75)

Animation examples are not very interesting here because using an atlas is OK, but there are use cases where each object could have orientations, so the idle animation would be intuitively constrained in one axis of the image. Imagine now having multiple animations of the same length in the same file for a single object for different orientations, it's OK with sequences and array textures, otherwise you need to fragment in multiple files; which is OK too, but you have now other alternatives.

The best of WebP to me is about lossy alpha; you can improve the compression of big alpha animations without having to make a choice between JPEG (no alpha) or PNG (lossless).

Edit: another example.
For HD isometric tiles like this (3.6 Mio).
Lossless WebP: 1.6 Mio (~55% reduction)Lossy WebP (q = 75): 452 kio (~87% reduction)

For 1 Go of this kind of resource (supposing similar compression), we could reduce it to 500 Mo while still using lossless compression. If we can afford loss, we could reduce it to 200 Mo.

[raidho36]

Video-ish image formats are not useful because texture atlases and texture arrays exist.

[Nelvin]

I can easily imagine games where such formats would be a decent option compared to regular animsheets.
Like all kinds of partialbackground/layer/item etc. animations for point & click adventures, visual novels, hidden object games etc. Of course you can use every frame as a decompressed one on texture sheets but in most cases, that's going to use a lot more resources.

[zorg]

Do note that the formats are only used for -storage- purposes; internally they'd be converted (or would need to be convertable to*):
- SoundData (raw samplepoint representation)
- Decoder (needing to wrap opus lib, would need to be able to realtime decode with some sensible buffer sizes... and would also necessitate to support SoundData-based encoding to fully make use of opus for transmission of recorded audio over the net, for example...; lib might also increase the size of the engine)
- ImageData (raw pixel values in some selected format, doesn't support animation - could need multiple ways on "depacking" frames into an atlas...)
- CompressedImageData (only includes GPU-formats like DXT1/5, BC5, 3Dc...)
- Texture (Image/Canvas) see above; ArrayTextures could hold anim frames though... constructors would become more complicated though
Also, while webp might be smaller than png, it's horridly more complex to manually edit yourself in any application that might not support video-based codecs like what webp uses... png is supported by almost anything... heck you could also just hex-edit it with a fair amount of success
But yeah, i'd agree with raidho since the game engine should just composit any animation from still images since it's capable of doing so without any issue; storing animations just feels like bloating games unnecessarily... then again, whatever people want i guess.

Speaking of, libmodplug's old and unmaintained, a better solution for those formats would be libopenmpt, if included library size doesn't matter much (pretty sure webp and opus' libs would be at least a few megs)

[Imagic]

There is complexity to benefit from all features offered by WebP (animation/sequence) or Opus (packet encoder/decoder).
At first, WebP and Opus could be added as regular formats like PNG or Vorbis, without API evolutions, it would still yield improvements.

Of course you can use every frame as a decompressed one on texture sheets but in most cases, that's going to use a lot more resources.

Do note that the formats are only used for -storage- purposes; internally they'd be converted

WebP animations could be implemented for Array textures loading (new API) and as a Video format without sound (streaming, as in a web browser). Formats have strengths and weaknesses, WebP and Opus are very versatiles additions to existing formats.

Also, while webp might be smaller than png, it's horridly more complex to manually edit yourself in any application that might not support video-based codecs like what webp uses... png is supported by almost anything... heck you could also just hex-edit it with a fair amount of success

File formats are not only made to be editable/readable. Working on a WebP file for a texture would be like working on a H.264 (or Theora) file for video editing. It's an export format.

(pretty sure webp and opus' libs would be at least a few megs)

On Windows, libopus.dll and libwebp.dll are around 300ko each. Probably smaller if statically linked.

[raidho36]

I don't believe library size has much to do with anything (see: box2d), it's the licensing and ability to statically link in general, and whether or not it's a duplicate feature.

A game engine is not a media player, it doesn't need to support all popular formats, in fact it doesn't needs to support any of them if it's made to only accept its own special format. You as a developer have the capacity to export your assets in any format, so there's no reason a game framework should support anything beyond its format of choice. And LOVE has very sensible, adequate and more or less industry standard choice of formats.

[Imagic]

WebP and Opus are not some bloatware random formats. They have been designed to be open standards and you can find how they perform with a simple research.

Anyway, I will experiment with LÖVE and probably do a pull request for a basic implementation.

[zorg]

Of course you can use every frame as a decompressed one on texture sheets but in most cases, that's going to use a lot more resources.

Do note that the formats are only used for -storage- purposes; internally they'd be converted

WebP animations could be implemented for Array textures loading (new API) and as a Video format without sound (streaming, as in a web browser). Formats have strengths and weaknesses, WebP and Opus are very versatiles additions to existing formats.

In my reading, this is what'd happen if webp support gets added:
- newImageData: Can't handle animated webp-s (maybe the lib's api could load the first frame only?)
- newImage: Can't handle animated webp-s (maybe the lib's api could load the first frame only?)
- newArrayImage: Could handle both animated and static webp-s (could pose a problem if you'd try to include both static images and an animated webp in the table you're supposed to give to this constructor...)
- newVideo: Needs new video decoder implementation that uses the webp lib's api... though i seem to recall video playback being synced to audio in löve or something (i might be wrong on that); if so, it'd be hard to do when no audio track exists.

That said, it's just a matter of the webp format getting inconsistent support in terms of what constructors can use what kind of webp files.

Edit: Do note that i agree in that the format itself is good if you want a format that's small in terms of storage space; i just don't know about the animation part of it.

[Nelvin]

Well IMO the actual value of such a format (if I don't get anything wrong about it in general) is that you for sure would not want to decode all the frames of an animation upfront into textures/texture atlases but on demand (code controlled) as you use the animation.

As already written, think of any kind of 2d adventure game (point & click, hidden object ...) where you can have some reasonably big and long animations (animated trees, charactersprites) with lots and lots of animation frames but you ever only need one of them in VRam. So you decode the sprites into a imageData and refresh it to it's actual texture. Does this cost other resources, like CPU and ram->vram bandwidth? Yes of course, nothing is free. But such games typically have next to nothing to do for a CPU anyway.

Does it need some additional convertion step in your asset pipline? Again, yes, for sure. But any project that's more than a trivial jam game needs an organized and automated pipeline anyway. You typically don't work with your ingame data in editors, i.e. if you want to do some modifications on an audio file you start with the uncompressed WAV, you don't load an ogg/mp3 and recompress it afterwards. The same goes for jpgs, and also for packed texture atlases if you want to change some sprites somewhere on the atlas etc.
So conversion is nothing but a small addition to your existing assetpipeline anyway and whether the tools you use to create/modify your source assets does not support the format is completely irrelevant.

[raidho36]

At that point there is no benefit to using animation-supporting format over any other format.