decklink: simplify the mode string usage

The mode can never be NULL. If it was, autodetect is set to false so we never
reach this code.

We can optimize the strlen() calls with a single strnlen() call that will only
test up to 5 chars to check we don't have 3 or 4 chars.