Skip to content

Draft: fix some `malloc(0)` platform inconsistency

Lyndon Brown requested to merge jnqnfe/vlc:malloc0 into master

If malloc() is given a size of zero, the result is implementation defined. Implementations may either return a null pointer or may give a non-null pointer that is only suitable for use with free() or possibly realloc().

(ref. https://wiki.sei.cmu.edu/confluence/display/c/MEM04-C.+Beware+of+zero-length+allocations)

There are many places in the codebase that already behave correctly and consistently with respect to this, but there are also many that do not, that allow malloc(0) calls to be made and then will react differently for the two different results. Such inconsistency is obviously undesirable.

I have assessed every use of malloc() throughout the entire codebase (excluding ./contrib) to find and fix in this commit the bulk of all cases that do not account for this implementation difference. I have skipped modifying any that have obvious conditions that prevent a zero size from reaching the malloc() calls. I have also skipped those where a zero may only result through calculation overflows (too much work for now to implement proper overflow handling).

Note that this work only covers malloc(). Additional work has also been done covering calloc(), realloc(), vlc_alloc(), etc separately, to be submitted after this.

Note that many places in the codebase questionably make no distinction between a null return for a zero sized allocation and one from an actual failed allocation, just treating both as an OOM condition. This may be worth a review in itself.

Note that fixes for the x264 codec and puzzle video filter plugins have been held back for now.

Edited by Lyndon Brown

Merge request reports