Skip to content
Commit 1bdb776c authored by David Conrad's avatar David Conrad
Browse files

Fix overflow in 8-bit NEON ADST

In 8-bit adst, it's possible that the final Round2(x[0], 12) can exceed
16-bits signed

Specifically, in 7.13.2.6. Inverse ADST4 process, the precision requirement is:
"It is a requirement of bitstream conformance that all values stored in the
s and x arrays by this process are representable by a signed integer using
r + 12 bits of precision."

For 8 bits, r is 16 for both row and column, so x[] can be 28-bit signed.
For values [134215680, 134217727] (within 2047 of the maximum 28-bit value),
the final Round2(x[0], 12) evaluates to 32768, exceeding 16-bits signed.

So switch to using sqrshrn, which saturates to 16-bits signed

This is a continuation of: Commit b53ff29d
arm: itx: Do clipping in all narrowing downshifts
parent 08c70801
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment