libvlc: Fail to render a video when using libvlc_media_player_set_xwindow()

Problem description

I'm trying to integrate libvlc in an application, and I use libvlc_media_player_set_xwindow() to request libvlc to render in my window. Note that I had previously no issue with a previous version of VLC (c1770ffc).

I'm trying to update to master (SHA-1 used for my tests: 79084ecb) and now I have a blocking issue. If I call libvlc_media_player_set_xwindow(), I have this log in loop:

[00007f4c9c050860] main video output debug: requested window size: 1024x576
[00007f4c9c050860] main video output debug: deinterlace -1, mode auto, is_needed 0
[00007f4c9c050860] main video output debug: Opening vout display wrapper
[00007f4c9c050860] main video output error: video output display creation failed
[00007f4c9c137e10] dbus_screensaver inhibit debug: got cookie 363065515
[000055e8ff631030] main player debug: saving a free vout
[h264 @ 0x7f4cbc06a740] get_buffer() failed
[h264 @ 0x7f4cbc06a740] thread_get_buffer() failed
[h264 @ 0x7f4cbc06a740] decode_slice_header error
[h264 @ 0x7f4cbc06a740] no frame!
[00007f4cbc02ba90] main decoder debug: vout change: decoder size
[00007f4cbc02ba90] main decoder debug: vout change: visible size
[00007f4cbc02ba90] main decoder debug: vout change: chroma

If I remove the call to libvlc_media_player_set_xwindow(), libvlc creates its own windows and there is no issue.

I managed to reproduced the same issue with doc/libvlc/gtk_player.c. Here is the small patch I used to fix the build with GTK3: gtk3_player.patch.

git bisect results

I tried a git bisect, and it says that the issue has been introduced in eea9f1f4, which has been introduced by !1220 (merged).

I made three tests:

• Revert only eea9f1f4: I still have the issue. But revert a single commit in a multi-commit MR may be a bad idea
• Revert all the changes introduced by !1220 (merged): it works
• Go back to 67ef3573 (first commit before !1220 (merged)): it works

assigned to @Courmisch

changed milestone to %4.0

added Component::Bindings: LibVLC (native API) label

added Severity::major Type::bug labels

tip: you shouldn't quote eea9f1f4 as code, so that it is correctly transformed into links. ;)

Woops :)

changed the description

I've taken a look, the problem is that this change:

-    if (splitter_name == NULL)
+    if (splitter_name == NULL) {
         modlist = modlistbuf = var_InheritString(vout, "vout");
-    else
+        if (unlikely(modlist == NULL))
+            return NULL;
+    } else
         modlist = "splitter,none";

...is incompatible with the fact that there are instances like var_SetString (p_mi, "vout", ""); setting the variable to an empty string.

(var_InheritString() will return a null for an empty string).

So to fix this, either:

1. That null check needs removing (the function the variable is passed to handles nulls so this is fine).
2. Or we need to change instances like var_SetString (p_mi, "vout", ""); to var_SetString (p_mi, "vout", "any");, but in that case for consistency we'd probably also want to do it for variables like "aout" and "window" as well, as would seem intended by the original (incomplete) MR.

Here is the small patch I used to fix the build with GTK3: gtk3_player.patch.

Please feel free to contribute these changes as well. Efforts to keep the samples building are always welcome.

closed with merge request !1327 (merged)

closed with commit 201eaaed

added Status::fixed label