vout_subpictures: fix bogus SPU scale check

And move + rename some code.

This fixes some odd SPU placement when the scale doesn't apply in one dimension.