decklink: don't return a whole es_format_t and split return types

And return an error when we can't use a format. We shouldn't pretend everything is OK.