I forgot what the original justification was for using short timeouts
here (perhaps some driver bug?) but in any case, it doesn't seem to make
any sense anymore.

This reduces CPU usage drastically on some drivers. I kept one loop at
10 ms since it shouldn't be hit anyways, and this way we know it's stuck
in that loop for absurd amounts of time.