renderer: perform some basic parameter validity checks

Fixes videolan/libplacebo#96 by
making it clear to API users that such invocations are not supported by
the API. (Probably doesn't fix the actual bug there, but that seems to
be downstream)