This is important both for stuff like debanding (which doesn't want the
output size to change) as well as stuff like upscaling (which *does*
want the output size to change).