decoder: fix forgotten vout error state

If the pool allocation fails or if the vout fails to start (via
input_resource_RequestVout()), an error is returned from
decoder_UpdateVideoFormat(). The next call to
decoder_UpdateVideoFormat() will return a success because the vctx will
be the same since only this variable is checked to detect a vout change.

To fix this issue, clean the owner->fmt and release the vctx when
returning an error.