Either the decoder is fast enough and everything works, either it is not
and there is no solution.

The frames are discarded once decoded if necessary, the decoder always
discard any previous frame. But do not discard blocks, that would
corrupt the video stream.