API change: convert x264_malloc() to size_t

Also prevents size overflows in PREALLOC(). Closes #10.

Good idea.

I am not a fan of types which size depends from compiling architecture (i.e. 32-bit vs 64-bit) like size_t. Their use should be limited to pointer arithmetic (intptr_t/uintptr_t) or when external API you call specify it. All other code should use fixed size types so that if it ok/fails/overflow for 32-bit architecture it should do same for 64-bit architecture and vice versa. As for singed/unsigned types my preference are signed types (where it is possible) because implicit conversion in C to unsigned types cause a lot of bugs that difficult to notice and debug (they can mask overflows and break comparisons). Also most of calculations in other places are done with int type so are already signed. So in my patch I decided to use int64_t for x264_malloc() and not size_t or uint64_t. Also I added explicit checks that size is not negative and not above SIZE_MAX.

Replaced by 7923c581 in master.

closed