Fix possible memory leak when x264_encoder_open() fails

In some scenarios, x264_encoder_open() can fail after it has allocated a x264_param_t structure, and is not freed before the function returns NULL to the caller. Call x264_param_cleanup() in that case.

Fixes #75, and replaces !168.