oss-fuzz: signed integer overflow in inv_adst8_1d
Steps to reproduce
- build with
-Db_sanitize=undefined
- run
./dav1d_fuzzer clusterfuzz-testcase-minimized-dav1d_fuzzer-5659429907726336
clusterfuzz-testcase-minimized-dav1d_fuzzer-5659429907726336
../../src/dav1d/src/itx_1d.c:685:35: runtime error: signed integer overflow: -752299 * 2896 cannot be represented in type 'int'
#0 0x4a5f01 in inv_adst8_1d /src/dav1d/src/itx_1d.c:685:35
#1 0x4a43b4 in inv_txfm_add_c /src/dav1d/src/itx_tmpl.c:0
#2 0x4a1e0a in inv_txfm_add_adst_dct_8x8_c /src/dav1d/src/itx_tmpl.c:129:1
#3 0x4c44b5 in dav1d_recon_b_intra_16bpc /src/dav1d/src/recon_tmpl.c:1049:33
#4 0x448636 in decode_b /src/dav1d/src/decode.c:1144:13
#5 0x43a67f in decode_sb /src/dav1d/src/decode.c:1925:17
#6 0x43b078 in decode_sb /src/dav1d/src/decode.c:1969:21
#7 0x43ab08 in decode_sb /src/dav1d/src/decode.c:1966:21
#8 0x43b078 in decode_sb /src/dav1d/src/decode.c:1969:21
#9 0x43995d in dav1d_decode_tile_sbrow /src/dav1d/src/decode.c:2378:13
#10 0x43deac in dav1d_decode_frame /src/dav1d/src/decode.c:2723:29
#11 0x441255 in dav1d_submit_frame /src/dav1d/src/decode.c:3108:20
#12 0x432c13 in dav1d_parse_obus /src/dav1d/src/obu.c:1130:20
#13 0x430ca8 in dav1d_decode /src/dav1d/src/lib.c:201:20
#14 0x42eb67 in LLVMFuzzerTestOneInput /src/dav1d/tests/libfuzzer/dav1d_fuzzer.c:101:19
#15 0x4e9eb8 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:571:15
#16 0x4da74d in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/libfuzzer/FuzzerDriver.cpp:280:6
#17 0x4de67b in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:713:9
#18 0x4da438 in main /src/libfuzzer/FuzzerMain.cpp:20:10
#19 0x7f0c8644082f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/libc-start.c:291
#20 0x405c28 in _start