Synchronising video and audio

[MachineCode]

This is probably a question for Zorg. I am generating video frames with audio to be played simultaneously. The audio will be queued based on 60hz frames from the draw function. So, given the sample rate, can I rely on the SR * 1/60 sample count to not slowly slip out of sync with the video frame rate? Unless the audio sample rate is locked to the video rate, then over a few minutes the audio could move out of sync.

If the audio system has a clock that is not derived from the video master clock, then based on XTAL accuracy of 20 ppm there will probably be drift. If that is the case, then rather than expensive interpolation, it might be OK to discard or duplicate a few samples each frame to keep in sync. I haven't explored this yet, but would be interested if anyone else has.

[zorg]

I mean, whatever you intend to do, OpenALSoft will use the configured Sample/ling Rate (let's say, 44100 smp/sec) to play back audio; whether you're sending less or more to the QSource will only alter whether underruns will happen or not. I usually just test whether it can accept any more data or not, and if it says it still has free buffers, i queue up the next SoundData buffer.

I can't say i ever tried to sync (actual) video with audio in löve before; custom "animations", sure. If i were you, i'd try small buffers for one, 1024-2048 smp-s long, and depending on how love.video works with drawing frames (how the frame index is calculated, i never touched that module so i don't know), you might either be able to nudge the video to display a frame for a bit longer, or less long; or, you might have to resort to what you said about discarding/duplicating samplepoints.

No idea where you got the 20 pulse per minute crystal accuracy from though.

[pgimeno]

You can perhaps keep the synchronization by examining the drift (in samples) and adjusting the pitch of the sources to match.

Edit: I believe ppm meant parts per million in this case.

[MachineCode]

Yes - sorry, ppm is a hw engineer term that means parts per million. Crystals are typically 20ppm, so even though that is very close to perfect, if the video frame rate and the audio sample rate are generated by different xtals, then you will get long term drift.

Suppose your sample rate is 44.1KHz. At 60 Hz that is 735 samples every frame. If you have 5 mins of video at 60hz and 5 mins of audio, then depending on the xtals that drive the video and the audio, you might be out by about 265 samples by 5mins. That is 1/3 of a video frame and for longer playback it could be a problem.

This may not be an issue if you are using HDMI audio as the audio clocks may be derived from the video master clock, and then no drift. I was thinking that if you can get the position of play in the buffer at draw time, you might be able to detect drift, however it would need some serious filtering to get rid of the jitter.

[pgimeno]

I was thinking that if you can get the position of play in the buffer at draw time, you might be able to detect drift, however it would need some serious filtering to get rid of the jitter.

I don't think so. If you're delayed, raise the pitch slightly; if you're advanced, lower it. Limit the maximum pitch change in both directions, so that it's as small as possible, and it should be OK.

In this thread, a similar technique is used in Mari0: http://www.love2d.org/forums/viewtopic.php?f=4&t=84769

[zorg]

Also, just to disagree with you a bit, i'm pretty sure that most computers only have between 1 and 3 actual crystal oscillators, and while everything uses those for clocking purposes, modern process-switched OS-es will mess up precise timing in any case; so i wouldn't even worry about such accuracy ever.

Your processor drives the video and audio rates, and if the decoders ṣ̵̖̙̝̝̰̺̳͉̮̈͌̑̒̈ͪͥ̽͋͟u̷̢͙̯̮̦̱͖̗̮̬̞̯͉̙͈͕̜̥ͭ͛̿ͫ͒͌̏́ͨ̊̓̓̆ͤ̐̈́̌̿͟͝c͉̤̣̲̞͎̫̘̥̮̲̺̩̻̰̙̓̃̔͌ͫ̽̐̾̄̊̀̄͒ͫͬ́͞ḵ̛̯̦̻̼͕͓͇̘̫̳͈̜̪̣̌̅̂ͬ͟͠ͅ ̛͕̙̘̲̦̙̲ͣ̉̇͆͋ͭ̑ͯͮͬ̒͜dͪ͐ͣ̔̓̀̇ͩ͏̢̺̝͙͕͍̺͔̹̻́͢ͅį̶̬̦͈̱͙̠̲̠̲̖̲̞̗̝̬̜͚̲͂̾͐̏̏͂ͨ̅̓ͬ̑̕c̘͍͍̺̬͕̼̞̗̦̖ͮ͒͋̄̉̈́̉ͭ̍̌̃͐͋ͣ͋͆́͢ͅk̵̗̪̜̦̰͎̯̮̫̯̑̉̔̔ͦ̉ͪ̂͗ͥ̐̀͟͢, there's not much you can do apart from trying to re-sync them.

[leiradel]

If you update video frames depending on the audio you'll get hiccups in the video output. At least on emulators, It's better to lock video frames to vsync and adjust the audio resampler on the fly to generate more/less audio frames depending if it's going ahead or lagging behind.

Unless the difference in the monitor and the emulated system frame rates are too big, this solution allows for a smooth video and audio without cracks to go with it.

I do this in my libretro frontend, here: https://github.com/leiradel/RASuite/blo ... o.cpp#L161

[pgimeno]

I've attached a proof of concept of the sync idea. It syncs the audio to the video so that there are 735 samples per frame on average. It assumes 60 Hz vsync rate; for other rates, the maximum frequency adjustment might not be enough.

It uses pure proportional control; I assume that with PID or maybe just PI control it may work much better.

Tested on Löve 0.9.1 and 0.10.2.

While preparing this PoC, I've noticed a bug in 0.10.2 that was not present in 0.9.1: the first time it loops, the loop is shorter than it should be by about 8,000-10,000 samples. The second time around it seems to work just fine. Printing the result of tell() shows this, and during the first loop there's a huge drift (in the above order) that takes a while to recover from.

[zorg]

(Note that i did assume either 0.11 or 0.10.2 with lpghatguy's qsource implementation; hackish stuff with :tell and either :seek or :setPitch will always be lower-resolution.)
Also, while i did mention testing such timings with Qsources and non-Video objects, i did assume that the OP was asking about Video objects, for which i have no idea how the frame rendering "callback" function works.

[MachineCode]

@pgimeno thanks for that. A PID with long time constant is a good solution provided that snd:setPitch is reasonably low overhead. I really don't know what the low level audio machinery looks like in a PC, but I suppose there is fine grain control over the sample rate so it's just changing a register or two.

In any audio system the DACs will be driven by a hw clock of some sort and they will have a hw FIFO feeding them. Audio samples can't be generated by any cpu (unless it is a DSP) because the timing is uncertain due to pipelining, out of order execution and so on. The quality of digital audio is dependant on a very low sample jitter rate which is measured in picoseconds.

@zorg take a USB sound stick as an example. It must have its own xtal since the USB connection only has two data lines. Its master clock must be independent of the system video clock. On the other hand, take an ARM SOC. Everything is in the chip, including audio, so probably the audio master is derived from the same xtal as the video. In that case, audio may not drift from the video frame rate. Most PC motherboards split the audio off onto a chip like a Realtek ALC665. I just looked on an old MB lying around, and there is a ALC665 with a 24.576MHz Xtal soldered next to it. 24.576MHz divided by 128 gives 192KHz exactly, which is probably the highest standard sample rate the chip can do. To generate other sample rates it will have internal phase locked loops using M/N integer ratio dividers. Either way, unless your computer is a SOC, then the audio sample rate will very likely derive from a different HW xtal.