After a certain input size, the gather kernel pulls ahead. This is
either due to interactions with the size of the texture cache, or due to
alignment with the 2x2 gather input grid, or both.

Regardless, this should generally speed things up for very high radius
polar kernels (especially when downscaling).