Commit 6a10a981 authored by Ronald S. Bultje's avatar Ronald S. Bultje

Rewrite inverse transforms to prevent integer overflows

The basic idea is that with intermediates of 19+sign bits and
multipliers of 12+sign bits, the intermediates are 19+12=31+sign
bits, and adding two of these together can overflow, which is UB
in C. These are not valid AV1 streams, but they are codable, and
so although we don't particularly care about the pixel-level
output for such streams, we do want to prevent triggering UB,
since that could be considered a security vulnerability.

To resolve this, we clip all multipliers to 11 bit by inverting
them:

(a * constant_1 + b * constant_2 + 2048) >> 12, where
constant_1 < 2048 but constant_2 >= 2048, is identical to:
((a * constant_1 + b * (4096 - constant_2) + 2048) >> 12) + b,
and 4096 - constant_2 < 2048. In other places, where both
constants are a multiple of 2, we can reduce the magnitude of
both and round/shift by 11 instead of 12.

Do this in dct4,8,16,32,64 as well as adst8,16. Also slightly
simplify the final phase of idct64_1d by moving the add/sub to
before the multiply.

The adst4 is rewritten to be shaped like a matrix-multiply, and
then use the same idea on all 4 multipliers in the matrix, since
the sum of all 4 multipliers is still under 4096 in all cases.

Fixes clusterfuzz-testcase-minimized-dav1d_fuzzer-5709759466962944,
credits to oss-fuzz. Also fixes #223.
parent 12dc2be0
Pipeline #3551 passed with stages
in 4 minutes and 42 seconds