VLC crashes early when run with libgmalloc(3) on Mac OS X Leopard

libgmalloc(3) is an interposing dynamic library which places heap allocations next to an unmapped guard page. It is useful in detecting the root cause of heap corruption bugs like buffer overflow and use after free.

When you run VLC with it, it crashes, making it useless for helping to diagnose these issues.

(gdb) set env DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib
(gdb) run
GuardMalloc: Allocations will be placed on 16 byte boundaries.
GuardMalloc:  - Some buffer overruns may not be noticed.
GuardMalloc:  - Applications using vector instructions (e.g., SSE or Altivec) should work.
GuardMalloc: GuardMalloc version 18
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0xc9f27d50
0x011c0da7 in -[VLCMain init] (self=0xc9f27d40, _cmd=0x9648b8b8) at intf.m:417
417	    o_about = [[VLAboutBox alloc] init];

(gdb) bt
[#0](https://code.videolan.org/videolan/vlc/-/issues/0)  0x011c0da7 in -[VLCMain init] (self=0xc9f27d40, _cmd=0x9648b8b8) at intf.m:417
[#1](https://code.videolan.org/videolan/vlc/-/issues/1)  0x965868ce in -[NSCustomObject nibInstantiate] ()
[#2](https://code.videolan.org/videolan/vlc/-/issues/2)  0x965672cd in -[NSIBObjectData instantiateObject:] ()
[#3](https://code.videolan.org/videolan/vlc/-/issues/3)  0x965669ee in -[NSIBObjectData nibInstantiateWithOwner:topLevelObjects:] ()
[#4](https://code.videolan.org/videolan/vlc/-/issues/4)  0x9655d0ec in loadNib ()
[#5](https://code.videolan.org/videolan/vlc/-/issues/5)  0x9655ca4d in +[NSBundle(NSNibLoading) _loadNibFile:nameTable:withZone:ownerBundle:] ()
[#6](https://code.videolan.org/videolan/vlc/-/issues/6)  0x9655c690 in +[NSBundle(NSNibLoading) loadNibFile:externalNameTable:withZone:] ()
[#7](https://code.videolan.org/videolan/vlc/-/issues/7)  0x9655c5ce in +[NSBundle(NSNibLoading) loadNibNamed:owner:] ()
[#8](https://code.videolan.org/videolan/vlc/-/issues/8)  0x011c062e in Run (p_intf=0xba832f08) at intf.m:183
[#9](https://code.videolan.org/videolan/vlc/-/issues/9)  0x000f6e8e in RunInterface (p_intf=0xba832f08) at interface/interface.c:234
[#10](https://code.videolan.org/videolan/vlc/-/issues/10) 0x000f6fc3 in intf_RunThread (p_intf=0xba832f08) at interface/interface.c:142
[#11](https://code.videolan.org/videolan/vlc/-/issues/11) 0x000e9b0b in libvlc_InternalAddIntf (p_libvlc=0xb1d17df8, psz_module=<value temporarily unavailable, due to optimizations>, b_block=1, b_play=1, i_options=0, ppsz_options=0x0) at libvlc-common.c:1095
[#12](https://code.videolan.org/videolan/vlc/-/issues/12) 0x000e85ad in VLC_AddIntf (i_object=0, psz_module=0x0, b_block=1, b_play=1) at libvlc.c:145
[#13](https://code.videolan.org/videolan/vlc/-/issues/13) 0x00004b92 in main (i_argc=1, ppsz_argv=0xbffff1d0) at vlc.c:218

10.5.2, Tried on a Macbook Pro and a Mac Pro.

There's a use after free condition.

Breakpoint 1, -[VLCMain init] (self=0xc26abd40, _cmd=0x9648b8b8) at intf.m:411
411	    if( _o_sharedMainInstance) {
(gdb) n
414	        _o_sharedMainInstance = [super init];
(gdb) n
417	    o_about = [[VLAboutBox alloc] init];
(gdb) n
418	    o_prefs = [[VLCPrefs alloc] init];
(gdb) c
Continuing.

Breakpoint 1, -[VLCMain init] (self=0xcaabad40, _cmd=0x9648b8b8) at intf.m:411
411	    if( _o_sharedMainInstance) {
(gdb) n
412	        [self dealloc];
(gdb) n
417	    o_about = [[VLAboutBox alloc] init];
(gdb) n

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0xcaabad50
0x011c0da7 in -[VLCMain init] (self=0xcaabad40, _cmd=0x9648b8b8) at intf.m:417
417	    o_about = [[VLAboutBox alloc] init];

o_about is a member of self, which just got freed. Assigning to it causes memory corruption.

diff --git a/modules/gui/macosx/intf.m b/modules/gui/macosx/intf.m
index 18381cb..e482346 100644
--- a/modules/gui/macosx/intf.m
+++ b/modules/gui/macosx/intf.m
@@ -1,3 +1,4 @@
+
 /*****************************************************************************
  * intf.m: MacOS X interface module
  *****************************************************************************
@@ -410,6 +411,7 @@ static VLCMain *_o_sharedMainInstance = nil;
 {
     if( _o_sharedMainInstance) {
         [self dealloc];
+        return _o_sharedMainInstance;
     } else {
         _o_sharedMainInstance = [super init];
     }

changed milestone to %0.9.0-test1

Fixed in [4a9105dd].

added Status::fixed label

closed