VDPAU/VAAPI priority issue

On my laptop (Linux, X11), whenever I attempt to play a video without specifying --dec-dev, I get no output:

$ ./vlc video.mp4
…
[000060400030fc90] vdpau_avcodec generic: Using OpenGL/VAAPI backend for VDPAU
[000060d000213bf0] gl gl error: Could not create interop
[000060d000306c90] gles2 gl error: Could not create interop
…

My graphic card is:

$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 630 (rev 04)

Analysis

OpenGL interop

The input format received by the OpenGL vout is VLC_CODEC_VDPAU_VIDEO_420 with decoder device VLC_DECODER_DEVICE_VDPAU.

To upload input to OpenGL textures, the interop is loaded according to the module priorities.

• interop_vdpau, with priority 2, is loaded first, but fails because the OpenGL extension "GL_NV_vdpau_interop" is not available.
• interop_vaapi, with priority 1, is tried, but fails (as expected) because the decoder device type is VLC_DECODER_DEVICE_VDPAU.

The problem is that VDPAU should not be loaded with backend VAAPI (Using OpenGL/VAAPI backend for VDPAU), but we should use VAAPI directly instead. But this is not as simple.

VDPAU vs VAAPI priority

ffmpeg_GetFormat() attempts to use formats in the order given by hwfmts:

    static const enum AVPixelFormat hwfmts[] =
    {
#ifdef _WIN32
        AV_PIX_FMT_D3D11VA_VLD,
        AV_PIX_FMT_DXVA2_VLD,
#endif
        AV_PIX_FMT_VAAPI,
        AV_PIX_FMT_VDPAU,
        AV_PIX_FMT_NONE,
    };

It correctly attempts to use VAAPI first.

The call to lavc_UpdateVideoFormat() initializes a vlc_decoder_device (the variableinit_device):

    for( size_t i = 0; hwfmts[i] != AV_PIX_FMT_NONE; i++ )
    {
// …
        vlc_decoder_device *init_device = NULL;
        msg_Dbg(p_dec, "trying format %s", dsc ? dsc->name : "unknown");
        if (lavc_UpdateVideoFormat(p_dec, p_context, hwfmt, swfmt, &init_device))
            continue; /* Unsupported brand of hardware acceleration */
// …

• lavc_UpdateVideoFormat()
• calls decoder_GetDecoderDevice()
• which calls dec->cbs->video.get_device(), in practice ModuleThread_GetDecoderDevice
• which calls vlc_decoder_device_Create()
• which loads a "decoder device" module:

    char *name = var_InheritString(o, "dec-dev");
    module_t *module = vlc_module_load(&priv->device, "decoder device", name,
                                    true, decoder_device_Open, &priv->device,
                                    window);

(if --dec-dev= is not explicitly passed, name is "any").

The loaded decoder device follows the VLC modules priority rules:

• vdpau has priority 100
• vaapi has priority 1 or 2

So the "vdpau" decoder device is loaded first.

As a consequence, in ffmpeg_GetFormat(), the decoder device will always be loaded according to the VLC modules priorities, regardless of the AV_PIX_FMT_* hwfmt. So concretely:

• AV_PIX_FMT_VAAPI is tried with decoder device of type VLC_DECODER_DEVICE_VDPAU: it necessarily fails
• AV_PIX_FMT_VDPAU is tried with decoder device of type VLC_DECODER_DEVICE_VDPAU: it works, but then fails in the interop

We could change the module priorities:

diff --git a/modules/hw/vaapi/decoder_device.c b/modules/hw/vaapi/decoder_device.c
index bde1e62115..b0159fa4be 100644
--- a/modules/hw/vaapi/decoder_device.c
+++ b/modules/hw/vaapi/decoder_device.c
@@ -249,15 +249,15 @@ Open(vlc_decoder_device *device, vout_window_t *window)
 }
 
 #if defined (HAVE_VA_X11)
-# define PRIORITY 2
+# define PRIORITY 201
 # define SHORTCUT "vaapi_x11"
 # define DESCRIPTION_SUFFIX "X11"
 #elif defined(HAVE_VA_WL)
-# define PRIORITY 2
+# define PRIORITY 201
 # define SHORTCUT "vaapi_wl"
 # define DESCRIPTION_SUFFIX "Wayland"
 #elif defined (HAVE_VA_DRM)
-# define PRIORITY 1
+# define PRIORITY 200
 # define SHORTCUT "vaapi_drm"
 # define DESCRIPTION_SUFFIX "DRM"
 #endif

But that way:

• AV_PIX_FMT_VAAPI is tried with decoder device of type VLC_DECODER_DEVICE_VAAPI: in practice, it works and fixes my problem, but if it fails…
• AV_PIX_FMT_VDPAU is tried with decoder device of type VLC_DECODER_DEVICE_VAAPI: it will necessarily fail

So the decoder device to choose must, in some way, depend on the associated avcodec format.