Then I don't know what else to try. Without some sort of error message or clarification from nvidia, I have no idea what parameter is wrong, where, and why, or what to do to fix it. It works fine on actual mesa. (Y'know, being a mesa extension and all...)
Maybe we can just blacklist support for this extension from the nvidia driver, but I'm not happy blacklisting things from drivers because it prevents them from being reasonably able to fix it in the future.
So, after extensive testing, I've come to the conclusion that the original issue here is an nvidia driver bug. The triggering circumstances seem to be the combination of surfaceless platform and the r8 texture format. Using rgba8 instead of r8 works fine, and using an x11 window instead of a surfaceless pbuffer also works fine.
I managed to create a small independent reproducer, which I may polish and upload to nvidia if I can be bothered. But for now, worked around it by just testing with rgba8 instead of r8.