transcode: use new vlc_encoder_operations structure

Add a wrapper if the operations are not defined to modify the location
where the previous pf_ callback pointers were used. Those wrappers will
be removed later.