Playing mp3 / m4a - Weird sound decimation / distortion issues

I can't quite work out exactly why this happens. Sometimes it happens, sometimes it doesn't. It's not like it starts out working fine and then you get distortion issues. More like you open an audio file with vlc and it sometimes it sounds decimated and distorted and sometimes it sounds perfectly fine. I've only really had this issue since VLC 2 and it is still prevalent in the current version. Even closing and reopening doesn't work. OSX Mountain Lion Latest - VLC latest (not beta)