Ogg demuxer misbehaves with zero length packets.
For example, with Ogg/Theora the the codec will produce a zero byte packet when nothing in the frame changes.
This mechanism can also be used for variable frame-rate streams by setting the container frame rate to a common-multiple then signalling drops when the video is not changing. ... but this is not exclusively a VFR feature and normal files will have these zero byte packets whenever things are completely still.
VLC currently appears to be ignoring the zero byte packets and losing sync.
Some examples: http://myrandomnode.dyndns.org:8080/~gmaxwell/cortest/dupes.ogv (this file should take 30 seconds to play, it's 30001 frames at 1000 fps, with most of the frames as drops)