UndefinedBehaviorSanitizer: undefined shift in filter_plane_rows_y() src/lf_apply.c
Found with commit 128715b5
This issue is hit frequently when fuzzing.
Steps to reproduce:
- build dav1d with UndefinedBehaviorSanitizer (-fsanitize=shift)
- replay testcase with
./dav1d_fuzzer testcase.ivf
src/lf_apply.c:92:44: runtime error: left shift of 65535 by 16 places cannot be represented in type 'int'
#0 0x6de9ce in filter_plane_rows_y src/lf_apply.c:92:44
#1 0x6de9ce in dav1d_loopfilter_sbrow_8bpc src/lf_apply.c:276
#2 0x64f2c1 in dav1d_filter_sbrow_8bpc src/recon.c:1459:9
#3 0x5456cf in dav1d_decode_frame src/decode.c:2673:25
#4 0x54f8c6 in dav1d_submit_frame src/decode.c:3041:20
#5 0x51e4a4 in dav1d_parse_obus src/obu.c:1110:20
#6 0x518e50 in dav1d_decode src/lib.c:201:20
#7 0x512ab0 in LLVMFuzzerTestOneInput tests/libfuzzer/dav1d_fuzzer.c:82:19
#8 0x5134e6 in main tests/libfuzzer/main.c:87:11
Edited by Tyson Smith