use FFmpeg enums instead of int

It's nicer in debuggers or even some IDE.

I used PixelFormat instead of AVPixelFormat because that's what we use everywhere. Maybe we can change that with newer FFmpeg requirements.