Skip to content

Stack buffer overflow on macOS when playing multi-channel audio

On macOS, when playing a video with more than two channels of audio using a CoreAudio device with more than 2 output channels, VLC crashes.

As far as I can tell from my debug build, something in the call to AudioUnitInitialize() in au_Initialize() causes the overflow.

Crashed Thread:        22

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Termination Reason:    Namespace SIGNAL, Code 6 Abort trap: 6
Terminating Process:   VLC [56641]

Application Specific Information:
stack buffer overflow
Thread 22 Crashed:
0   libsystem_kernel.dylib        	       0x1939c4744 __pthread_kill + 8
1   libsystem_pthread.dylib       	       0x1939fbc28 pthread_kill + 288
2   libsystem_c.dylib             	       0x193909b74 __abort + 128
3   libsystem_c.dylib             	       0x1938fa640 __stack_chk_fail + 96
4   libauhal_plugin.dylib         	       0x107171a2c au_Initialize + 2380
5   libsystem_pthread.dylib       	       0x1939f6da0 thread_start + 8

This happens on both ARM64 and Intel. VLC versions 3.0.17.3 and 3.0.16 do not have the problem, but 3.0.18, 3.0.19 and 3.0.20 all do.

Steps to reproduce:

  1. Install the BlackHole 16 channel CoreAudio driver (can be downloaded from https://existential.audio/blackhole/ or built from https://github.com/ExistentialAudio/BlackHole)
  2. Download this MP4 AAC file with 5.1 sound channels: https://archive.org/download/5.1SurroundSoundTestFilesVariousFormatsAACAC3MP4DTSWAV/5.1%20Surround%20Sound%20AAC%20Test.mp4
  3. Play the MP4 file with VLC. If the BlackHole output is chosen as the default system output, VLC immediately crashes.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information