Stack buffer overflow on macOS when playing multi-channel audio
On macOS, when playing a video with more than two channels of audio using a CoreAudio device with more than 2 output channels, VLC crashes.
As far as I can tell from my debug build, something in the call to AudioUnitInitialize() in au_Initialize() causes the overflow.
Crashed Thread: 22
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Termination Reason: Namespace SIGNAL, Code 6 Abort trap: 6
Terminating Process: VLC [56641]
Application Specific Information:
stack buffer overflow
Thread 22 Crashed:
0 libsystem_kernel.dylib 0x1939c4744 __pthread_kill + 8
1 libsystem_pthread.dylib 0x1939fbc28 pthread_kill + 288
2 libsystem_c.dylib 0x193909b74 __abort + 128
3 libsystem_c.dylib 0x1938fa640 __stack_chk_fail + 96
4 libauhal_plugin.dylib 0x107171a2c au_Initialize + 2380
5 libsystem_pthread.dylib 0x1939f6da0 thread_start + 8
This happens on both ARM64 and Intel. VLC versions 3.0.17.3 and 3.0.16 do not have the problem, but 3.0.18, 3.0.19 and 3.0.20 all do.
Steps to reproduce:
- Install the BlackHole 16 channel CoreAudio driver (can be downloaded from https://existential.audio/blackhole/ or built from https://github.com/ExistentialAudio/BlackHole)
- Download this MP4 AAC file with 5.1 sound channels: https://archive.org/download/5.1SurroundSoundTestFilesVariousFormatsAACAC3MP4DTSWAV/5.1%20Surround%20Sound%20AAC%20Test.mp4
- Play the MP4 file with VLC. If the BlackHole output is chosen as the default system output, VLC immediately crashes.