Out-of-bounds read for RGB input with AVX2
When AVX2 is enabled, I get a segfault in x264_8_plane_copy_deinterleave_rgb_avx2
if my RGB input is not padded.
In the following, my input buffer goes from 0x7ffff5f22000
to 0x7ffff61c5000
. A backtrace and disassembly gives
(gdb) bt
#0 0x00007ffff7e073f2 in x264_8_plane_copy_deinterleave_rgb_avx2 () from /lib/x86_64-linux-gnu/libx264.so.160
#1 0x0000000000000000 in ?? ()
(gdb) disas $pc-20,$pc+20
Dump of assembler code from 0x7ffff7e073de to 0x7ffff7e07406:
0x00007ffff7e073de <x264_8_plane_copy_deinterleave_rgb_avx2+62>: mov %r11,%r14
0x00007ffff7e073e1 <x264_8_plane_copy_deinterleave_rgb_avx2+65>: vmovdqu 0x0(%rbp),%xmm0
0x00007ffff7e073e6 <x264_8_plane_copy_deinterleave_rgb_avx2+70>: vinserti128 $0x1,0xc(%rbp),%ymm0,%ymm0
0x00007ffff7e073ed <x264_8_plane_copy_deinterleave_rgb_avx2+77>: vmovdqu 0x18(%rbp),%xmm1
=> 0x00007ffff7e073f2 <x264_8_plane_copy_deinterleave_rgb_avx2+82>: vinserti128 $0x1,0x24(%rbp),%ymm1,%ymm1
0x00007ffff7e073f9 <x264_8_plane_copy_deinterleave_rgb_avx2+89>: vpshufb %ymm3,%ymm0,%ymm0
0x00007ffff7e073fe <x264_8_plane_copy_deinterleave_rgb_avx2+94>: vpshufb %ymm3,%ymm1,%ymm1
0x00007ffff7e07403 <x264_8_plane_copy_deinterleave_rgb_avx2+99>: vpblendd $0x22,%ymm1,%ymm0,%ymm2
End of assembler dump.
(gdb) p $rbp
$1 = (void *) 0x7ffff61c4fd0
If I pad the buffer, or if I disable CPU optimizations, the problem goes away.
After receiving help in #x264 on Freenode, I was informed that input/input.c
pads its input buffer, and that this can in particular be needed for RGB deinterleaving.
If the issue cannot be fixed, it would at least be helpful if the documentation could be updated with padding (and alignment?) requirements for the input buffers.
Thanks a lot to JEEB, BugMaster and Gramner on Freenode for helping me!