VLC does not play video after recovering from VDPAU preemption.

Below are the details of the bug.

1. Operating system – “Ubuntu 20.04.3 LTS”

2. VLC version – “VLC version 4.0.0-dev”

3. HW – “Any Nvidia driver”

4. Logs – vlc-log.txt

5. Steps to reproduce the problems:

    a.	Run the vlc using the command ./vlc /”video clip” --vout=xcb_x11
    b.	Do a VT switch out using the command: “sudo chvt 6”. In my case, 7 is the active terminal, hence "sudo chvt 6" is switching out from active terminal no 7.
    c.	Do a VT switch in using the command: “sudo chvt 7”. Since 7 is the active terminal, this command switches in to active terminal 7 from terminal no 6.
    d.	The black screen is seen and no video is played.
    e.	Expected behavior: The video should be seen playing.

6. Reproducibility – This issue is reproduced frequently.

7. Format of the file – Not required.

8. Backtrace – Bug is not a crash, hence the attachment of bug report is not required.

9. Video file – Not required.

10. Email-id - sjambekar@nvidia.com

Scenario:

After running the vlc command as explained in point no 5a and after switching out from the terminal, and after switching in again, the video is not played. As seen in the logs, the vdpau driver returns "invalid handle: 3" error. This is the behavior as of now. The expected behavior should be the video should start playing after recovering from preemption.

Analysis:

After going through the vlc code and after analysing the code by running the vlc application in the gdb, below is what I observed:

Investigation in the vlc application so far:

1. I see that the vo (video output) thread is dependent on the decoder thread for data, which is obvious.

2. However, for some reason, even after switching out and switching in, the function in vo thread is dependent on the vdpau decode call. Basically, there is a callback happening from vo thread to the VideoExport_Filter() in the hw/vdpau/chroma.c file. After switching out (where the hw decoder is destroyed) and switching in, ideally the vlc sw decoder should start decoding. The callback from the vo thread, should now call the decode function in the sw decoder path and not the hw decoder path.

3. Since the call back function in the vo thread is calling the hw decoder function, and since the hw decoder is no more active after switching out, the hw decoder function is rightly returning the invalid handle (err: 3).

What I think should happen:

1. In the vo thread, the correct callback function (sw decode or hw decode) should be called based on the pre-emption check.