[rub.de/24] Assertion failure in `SetupAudioES` (MP4 demuxing / ES setup)

Ref: https://oss-fuzz.com/testcase-detail/5317582524121088

Issue from rub.de: bug may be publicly disclosed on Tuesday, April 29, 2025.

MP4 files with an esds box in a soun stsd entry are rejected with an assertion (instead of erroring normally) in SetupAudioES in modules/demux/mp4/essetup.c if the entry does not report the sample type as mp4a (the assertion is i_sample_type == ATOM_mp4a).

xeon ~/work/git/vlc/build-asan $ gdb --args ./test/vlc-demux-dec-run /home/tom/Downloads/reproducer.mp4 
GNU gdb (Debian 15.2-1) 15.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test/vlc-demux-dec-run...
(gdb) r
Starting program: /home/tom/work/git/vlc/build-asan/test/vlc-demux-dec-run /home/tom/Downloads/reproducer.mp4
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
vlc-demux-dec-run: ../../modules/demux/mp4/essetup.c:1259: SetupAudioES: Assertion `i_sample_type == ATOM_mp4a' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, 
    no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, 
    no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007ffff709dcef in __pthread_kill_internal (threadid=<optimized out>, signo=6)
    at ./nptl/pthread_kill.c:78
#2  0x00007ffff7049c42 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff70324f0 in __GI_abort () at ./stdlib/abort.c:79
#4  0x00007ffff7032418 in __assert_fail_base (
    fmt=0x7ffff71b6ca0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=assertion@entry=0x7ffff49da620 "i_sample_type == ATOM_mp4a", 
    file=file@entry=0x7ffff49d91a0 "../../modules/demux/mp4/essetup.c", 
    line=line@entry=1259, 
    function=function@entry=0x7ffff49da8a0 <__PRETTY_FUNCTION__.1> "SetupAudioES")
    at ./assert/assert.c:94
#5  0x00007ffff7042552 in __assert_fail (
    assertion=assertion@entry=0x7ffff49da620 "i_sample_type == ATOM_mp4a", 
    file=file@entry=0x7ffff49d91a0 "../../modules/demux/mp4/essetup.c", 
    line=line@entry=1259, 
    function=function@entry=0x7ffff49da8a0 <__PRETTY_FUNCTION__.1> "SetupAudioES")
    at ./assert/assert.c:103
#6  0x00007ffff4992834 in SetupAudioES (p_demux=p_demux@entry=0x511000007480, 
    p_track=p_track@entry=0x517000003880, p_sample=p_sample@entry=0x50b000069080, 
    p_fmt=p_fmt@entry=0x5170000038a0, p_cfg=<optimized out>)
    at ../../modules/demux/mp4/essetup.c:1259
#7  0x00007ffff496c833 in TrackFillConfig (p_demux=p_demux@entry=0x511000007480, 
    p_track=p_track@entry=0x517000003880, p_sample=p_sample@entry=0x50b000069080, 
    i_chunk=i_chunk@entry=0, p_fmt=p_fmt@entry=0x5170000038a0, 
    p_cfg=p_cfg@entry=0x7ffff4e5ad20) at ../../modules/demux/mp4/mp4.c:3184
#8  0x00007ffff4977660 in TrackCreateES (p_demux=p_demux@entry=0x511000007480, 
    p_track=p_track@entry=0x517000003880, i_chunk=<optimized out>, pp_es=<optimized out>)
    at ../../modules/demux/mp4/mp4.c:3246
#9  0x00007ffff497dfc7 in MP4_TrackSetup (p_demux=p_demux@entry=0x511000007480, 
    p_track=0x517000003880, p_box_trak=<optimized out>, 
    b_create_es=b_create_es@entry=true, b_force_enable=<optimized out>)
    at ../../modules/demux/mp4/mp4.c:4045
#10 0x00007ffff49800ca in Open (p_this=<optimized out>)
    at ../../modules/demux/mp4/mp4.c:1310
#11 0x00007ffff7318aed in demux_Probe (func=0x7ffff497e3ef <Open>, forced=false, 
    ap=ap@entry=0x7ffff4f838c0) at ../../src/input/demux.c:112
#12 0x00007ffff72d7532 in vlc_module_load (log=<optimized out>, 
    capability=capability@entry=0x7ffff747e120 "demux", name=<optimized out>, 
    name@entry=0x55555555a200 "any", strict=strict@entry=false, 
    probe=probe@entry=0x7ffff7318a79 <demux_Probe>) at ../../src/modules/modules.c:230
#13 0x00007ffff731989c in demux_NewAdvanced (p_obj=p_obj@entry=0x511000006bc0, 
    p_input=p_input@entry=0x0, module=module@entry=0x55555555a200 "any", 
    url=url@entry=0x55555555a240 "vlc://nop", s=s@entry=0x511000006bc0, 
    out=out@entry=0x50300004da10, b_preparsing=<optimized out>)
    at ../../src/input/demux.c:196
#14 0x00007ffff7319bab in demux_New (p_obj=p_obj@entry=0x511000006bc0, 
    module=module@entry=0x55555555a200 "any", url=url@entry=0x55555555a240 "vlc://nop", 
--Type <RET> for more, q to quit, c to continue without paging--
    s=s@entry=0x511000006bc0, out=out@entry=0x50300004da10) at ../../src/input/demux.c:77
#15 0x00005555555582fc in demux_process_stream (args=args@entry=0x7ffff4d00020, 
    s=s@entry=0x511000006bc0) at ../../test/src/input/demux-run.c:294
#16 0x0000555555558669 in vlc_demux_process_url (args=args@entry=0x7ffff4d00020, 
    url=url@entry=0x504000000050 "file:///home/tom/Downloads/reproducer.mp4")
    at ../../test/src/input/demux-run.c:348
#17 0x0000555555558708 in vlc_demux_process_path (args=args@entry=0x7ffff4d00020, 
    path=0x7fffffffde73 "/home/tom/Downloads/reproducer.mp4")
    at ../../test/src/input/demux-run.c:362
#18 0x000055555555755a in main (argc=2, argv=0x7fffffffda08)
    at ../../test/vlc-demux-run.c:50

Sample: reproducer clusterfuzz-testcase-minimized-vlc-demux-dec-libfuzzer-5317582524121088