Audio is lost or is not immediately decoded when starting playback (first blocks are always late)

Whenever you start playing a file, the first few "buffers" of audio are lost. This may lead to cut off in the beginning, especially when a song starts immediately. What I noticed is when playing a MP3, 2 "buffers" are lost while when playing a FLAC, 1 "buffer" is lost.

Picture #2 (closed) (yes, I saved them in a different order) shows the "buffers" lost using VLC's built-in "Media Information". Picture #1 (closed) highlights the audio that is "lost" and not played when using VLC to play it.