clock: audio/video synchronisation issues

Following the discussion, we had in #26825 (closed)

The audio clock is only updated when playing new samples.

Clock points will largely depend on the input:

• one audio sample of 1sec every 1seconds
• 300ms burst of 20ms audio sample and a pause (buffering)

We can't control how the input/buffering will send audio samples.

Here is a graph that show a classic situation, where audio is played in burst (exacerbated by using a BT device):

src/clock/clock.c
/* Don't send double, but coeff as vlc_tick_t to get the same scale than audio pts */
VLC_TRACE("coeff", (vlc_tick_t) ((1 / instant_coeff) * 1000000)),                  
VLC_TRACE("coeff_avg", (vlc_tick_t) ((1 / main_clock->coeff) * 1000000)), 

The instant coeff vary from 0.0 to 3.0 before stabilizing around 1.0 (the playing rate). We can see that the average coeff take around 4 seconds to stabilize (hence the clock offset that is calculated from it).

Therefore, in our actual clock/aout implementation, the clock offset precision will largely depend on the audio sample size: the smaller is the audio sample, the more the coefficient will vary (high fps video will likely have smaller audio samples).

Our long term average calculation is not a good choice in that situation. It could only work if we send audio samples regularly (It's a good choice for vouts, 60fps => clock update every 16.6ms for example).

The audio clock is not updated soon enough

Due to buffering and how VLC input are implemented (may depends on access/demux modules), audio is sent in burst and the InputThread will wait regularly in order to not overflow decoders with data.

There is a problem if the first aout delay can be calculated while the InputThread is waiting. The first clock update will be delayed, sometimes, after the vout is started, resulting on video frame drops.

Clock offset measurement in most common situation:

All audio outputs show the same kind of issues.

PulseAudio with a BT device, playing a 25fps video (23ms audio sample)

PulseAudio with a BT device, playing a 60fps video (10ms audio sample)

PulseAudio with a speaker, playing a 25fps video (23ms audio sample)

PulseAudio with a speaker, playing a 60fps video (10ms audio sample)