Always initializes VAAPI/VDPAU with --vout=gl
I've tested VLC 2.2.8 and 4.0.0-dev-1566-g34b68b13, the behavior is in both.
This is mostly a nitpick, so I'm setting a super low priority. This issue mostly just causes user confusion over whether VAAPI/VDPAU is actually being used for hardware decoding.
If I specify --vout=gl --avcodec-hw=none, I would expect that neither VDPAU nor VAAPI get touched at all by VLC. This is the case with a different video output like --vout=x11_xcb --avcodec-hw=none. But when --vout=gl --avcodec-hw=none is specified, it initializes VDPAU or VAAPI. On an NVIDIA box it'll initialize VDPAU, on a box with Intel intgrated graphics it'll initialize VAAPI. It doesn't appear to actually use either of them for hardware decoding, but it does initialize them, unlike with --vout=xcb_x11.
First of all here's the expected behavior:
$ vlc --avcodec-hw=none --vout=xcb_x11 <filename>
VLC media player 4.0.0-dev Otto Chriek (revision 4.0.0-dev-1566-g34b68b13fa)
[0000000000a413f0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[0000000000b33c50] vlcpulse audio output error: digital pass-through stream connection failure: Not supported
[0000000000b33c50] main audio output error: module not functional
[00007fd1f0d22620] main decoder error: failed to create audio output
[0000000000b33c50] vlcpulse audio output error: digital pass-through stream connection failure: Not supported
[0000000000b33c50] main audio output error: module not functional
[00007fd1f0d22620] main decoder error: failed to create audio output
^CQObject::~QObject: Timers cannot be stopped from another thread
As expected, neither VAAPI nor VDPAU are initialized.
And here's the expected behavior when --avcodec-hw=vaapi:
$ vlc --avcodec-hw=vaapi --vout=xcb_x11 <filename>
VLC media player 4.0.0-dev Otto Chriek (revision 4.0.0-dev-1566-g34b68b13fa)
[00000000008cb3f0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[00000000009bdc50] vlcpulse audio output error: digital pass-through stream connection failure: Not supported
[00000000009bdc50] main audio output error: module not functional
[00007f1a0cd223f0] main decoder error: failed to create audio output
[00000000009bdc50] vlcpulse audio output error: digital pass-through stream connection failure: Not supported
[00000000009bdc50] main audio output error: module not functional
[00007f1a0cd223f0] main decoder error: failed to create audio output
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
[00007f1a0cc184a0] avcodec decoder: Using Intel i965 driver for Intel(R) Skylake - 2.1.0 for hardware decoding
[00007f19dc003180] main filter error: Failed to create video converter
^CQObject::~QObject: Timers cannot be stopped from another thread
So it successfully initialized VAAPI and used it (as noted by the avcodec decoder: line, also as expected.
This changes when --vout=gl is specified instead (this is the bug):
$ vlc --avcodec-hw=none --vout=gl <filename>
VLC media player 4.0.0-dev Otto Chriek (revision 4.0.0-dev-1566-g34b68b13fa)
[00000000015253f0] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
[0000000001617c50] vlcpulse audio output error: digital pass-through stream connection failure: Not supported
[0000000001617c50] main audio output error: module not functional
[00007fce60d22580] main decoder error: failed to create audio output
[0000000001617c50] vlcpulse audio output error: digital pass-through stream connection failure: Not supported
[0000000001617c50] main audio output error: module not functional
[00007fce60d22580] main decoder error: failed to create audio output
libva info: VA-API version 1.1.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_1
libva info: va_openDriver() returns 0
^C[h264 @ 0x7fce60cea0a0] get_buffer() failed
[h264 @ 0x7fce60cea0a0] thread_get_buffer() failed
[h264 @ 0x7fce60cea0a0] decode_slice_header error
[h264 @ 0x7fce60cea0a0] no frame!
[h264 @ 0x7fce60d062e0] get_buffer() failed
[h264 @ 0x7fce60d062e0] thread_get_buffer() failed
[h264 @ 0x7fce60d062e0] decode_slice_header error
[h264 @ 0x7fce60d062e0] no frame!
QObject::~QObject: Timers cannot be stopped from another thread
Note that although it initialized VAAPI, it didn't actually appear to enable it for hardware decoding (no avcodec decoder: line). I would expect it to respect --avcodec-hw=none as --vout=xcb_x11 does, and not load/initialize libva at all.