signal 11 (SIGSEGV), code 2 (SEGV_ACCERR) crash on Android 12 arm64 devices

ffmpeg cross-compiled with x264 for Android arm64 architecture crashes when it is run on Android 12 (Api Level 31) devices.

Don't know if it happens because of an invalid compile flag, setting, etc. Here is what we learned during our tests:

• This case doesn't occur if x264 is built without asm.
• Command used to reproduce the crash uses 792x1568 dimensions. Not all dimensions cause a crash. 792x1568 is one of the dimension that does.
• Older Android devices or emulators e.g. Android 11 (Api Level 30) doesn't have this issue. Android 11 have some new changes regarding ARM Memory Tagging Extension (MTE). Maybe those changes are triggering it. See Tagged Pointers for those changes.

This is the ffmpeg command used to reproduce this case.

ffmpeg -v 9 -loglevel 99 -loop 1 -i pyramid.jpg -vf scale=792x1568 -c:v libx264 video.mp4

Full console output is available at console-output.txt file.

Disassembly information captured via gdb can be seen inside disassembly.txt.

Below you can see the tombstone file generated.

    Build fingerprint: 'google/sdk_gphone64_arm64/emulator64_arm64:12/SE1A.220630.001/8789670:userdebug/dev-keys'
    Revision: '0'
    ABI: 'arm64'
    Timestamp: 2022-08-28 11:31:51.574993028+0100
    Process uptime: 1s
    Cmdline: ./ffmpeg -v 9 -loglevel 99 -loop 1 -i pyramid.jpg -vf scale=792x1568 -c:v libx264 video.mp4
    pid: 31986, tid: 31986, name: ffmpeg  >>> ./ffmpeg <<<
    uid: 0
    tagged_addr_ctrl: 0000000000000001
    signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb400007a2a46b000
        x0  b400007b5d718960  x1  b400007a2a420cd0  x2  b400007a2a425904  x3  b400007a2a433ee4
        x4  b400007a2a46aff4  x5  0000007fc9bbd8ec  x6  00000000fffffffa  x7  0000000000000032
        x8  b400007a2a4232b0  x9  b400007a2a4689b0  x10 0000007cdd03e738  x11 00000000000025e4
        x12 0000000000008000  x13 0000000000000031  x14 0000000000000061  x15 0000000000007fff
        x16 b400007a2a433e74  x17 b400007a2a3ff3a8  x18 0000007cee20c000  x19 0000000000000027
        x20 0000007fc9bbeb28  x21 0000000000000027  x22 00000000000012f2  x23 b400007ced0d6dd0
        x24 b400007a2a431890  x25 b400007a2aa45c60  x26 b400007a2a433e74  x27 b400007b5d718900
        x28 b400007a2a420c60  x29 0000007fc9bbd8f0
        lr  0000007cdcffad9c  sp  0000007fc9bbd8a0  pc  0000007cdd03e74c  pst 0000000080001000

    backtrace:
        #00 pc 000000000060c74c  /data/local/tmp/org.ffmpeg.test/libavcodec.so (x264_8_mbtree_propagate_cost_neon+20)
        #01 pc 00000000005c8d98  /data/local/tmp/org.ffmpeg.test/libavcodec.so (macroblock_tree_propagate+540)
        #02 pc 00000000005bd9b4  /data/local/tmp/org.ffmpeg.test/libavcodec.so (macroblock_tree+856)
        #03 pc 00000000005bcef0  /data/local/tmp/org.ffmpeg.test/libavcodec.so (x264_8_slicetype_analyse+2844)
        #04 pc 000000000060f4c4  /data/local/tmp/org.ffmpeg.test/libavcodec.so (x264_8_lookahead_get_frames+264)
        #05 pc 00000000005962ac  /data/local/tmp/org.ffmpeg.test/libavcodec.so (x264_8_encoder_encode+896)
        #06 pc 0000000000592360  /data/local/tmp/org.ffmpeg.test/libavcodec.so (x264_encoder_encode+16)
        #07 pc 000000000058c714  /data/local/tmp/org.ffmpeg.test/libavcodec.so (X264_frame+1272)
        #08 pc 00000000004838b8  /data/local/tmp/org.ffmpeg.test/libavcodec.so (ff_encode_encode_cb+36)
        #09 pc 0000000000483db0  /data/local/tmp/org.ffmpeg.test/libavcodec.so (encode_receive_packet_internal+340)
        #10 pc 0000000000483c10  /data/local/tmp/org.ffmpeg.test/libavcodec.so (avcodec_send_frame+516)
        #11 pc 0000000000035f14  /data/local/tmp/org.ffmpeg.test/ffmpeg (encode_frame+252)
        #12 pc 0000000000035950  /data/local/tmp/org.ffmpeg.test/ffmpeg (do_video_out+1840)
        #13 pc 0000000000034eec  /data/local/tmp/org.ffmpeg.test/ffmpeg (reap_filters+280)
        #14 pc 000000000003129c  /data/local/tmp/org.ffmpeg.test/ffmpeg (main+7160)
        #15 pc 00000000000488c8  /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init+96) (BuildId: ba489d4985c0cf173209da67405662f9)

tombstone.txt includes the full report.

Steps we used to build and run ffmpeg and x264 can be found under the android-ffmpeg-x264-arm64-api-level-31 repository.