Abort frame decoding properly on reference error

This could cause a frame waiting on the current one to not be notified on error.

Fixes #351 (closed).