avcodec: prioritize RGB with matching mask

The VLC_RGB macro maps the RGB with mask to/from VLC formats. But it also maps one of the FFmpeg variants with no mask at all. That means it might be picked even though there's another format with exactly the proper mask.