x264 was incorrectly calculating cpb_removal_delay with respect to the first keyframe.
It should have been calculating cpb_removal_delay with respect to the last keyframe.