I don't know if this bug still affects latest x264, but it can't hurt to try to detect it.
Accordingly refuse to open the encoder if detected.
Apparently VLC (on Windows) has been distributed for some time with a completely
broken x264 due to the use of a completely broken compiler (gcc 4.2).  In
particular, the MV costs seem to be calculated incorrectly on win32 when linking
from an application compiled without -ffast-math to an application with
-ffast-math.
I am not entirely certain why this occurs, but the result is, unsurprisingly,
encoding quality that makes MPEG-2 look good, due to the motion search being
completely broken.