coreaudio: fix stack overflow in MapOutputLayout()

Fixes #28424 (closed)

This commit need to be backported to VLC 3.0.

Note that the crash was a regression, but the output channel mismatch was always there, cf. !4570