Similar to the previous commit, we need to ensure compatibility with the
handle type we are interested in to avoid UB.

Also rewrite the image format compatibility check logic in general to
make it more future-proof. Also fixes a few memleaks on error (due to
`return NULL` when it should have done `goto error`).