x86: Fix overflows in 12bpc AVX2 identity itx asm

12bpc has 20-bit coefficients, which can't be multiplied by factors larger than 4096 without int32_t overflows. The exception is for 1:2 and 2:1 sizes, which can handle multiplies up to 5793 after the initial sqrt(2) scaling.