libvlc: Fail to render a video when using libvlc_media_player_set_xwindow()

Problem description

I'm trying to integrate libvlc in an application, and I use libvlc_media_player_set_xwindow() to request libvlc to render in my window. Note that I had previously no issue with a previous version of VLC (c1770ffc).

I'm trying to update to master (SHA-1 used for my tests: 79084ecb) and now I have a blocking issue. If I call libvlc_media_player_set_xwindow(), I have this log in loop:

[00007f4c9c050860] main video output debug: requested window size: 1024x576
[00007f4c9c050860] main video output debug: deinterlace -1, mode auto, is_needed 0
[00007f4c9c050860] main video output debug: Opening vout display wrapper
[00007f4c9c050860] main video output error: video output display creation failed
[00007f4c9c137e10] dbus_screensaver inhibit debug: got cookie 363065515
[000055e8ff631030] main player debug: saving a free vout
[h264 @ 0x7f4cbc06a740] get_buffer() failed
[h264 @ 0x7f4cbc06a740] thread_get_buffer() failed
[h264 @ 0x7f4cbc06a740] decode_slice_header error
[h264 @ 0x7f4cbc06a740] no frame!
[00007f4cbc02ba90] main decoder debug: vout change: decoder size
[00007f4cbc02ba90] main decoder debug: vout change: visible size
[00007f4cbc02ba90] main decoder debug: vout change: chroma

If I remove the call to libvlc_media_player_set_xwindow(), libvlc creates its own windows and there is no issue.

I managed to reproduced the same issue with doc/libvlc/gtk_player.c. Here is the small patch I used to fix the build with GTK3: gtk3_player.patch.

git bisect results

I tried a git bisect, and it says that the issue has been introduced in eea9f1f4, which has been introduced by !1220 (merged).

I made three tests:

• Revert only eea9f1f4: I still have the issue. But revert a single commit in a multi-commit MR may be a bad idea
• Revert all the changes introduced by !1220 (merged): it works
• Go back to 67ef3573 (first commit before !1220 (merged)): it works