VLC Super Resolution Memory Leak

added Component::Video: Direct3D / Win32 label

assigned to @robUx4

I can't reproduce the issue with this build https://artifacts.videolan.org/vlc-3.0/nightly-win64/20230509-0220/

Can you provide some debug logs to see what's going on ? You may also reset your preferences to default in case you have other filters doing things.

changed milestone to %3.0.x maintenance

I tried with the build you mention, but still the same issue. Here the debug logs : log_RTX_Debug

I found that there's no issue with this build : https://downloads.videolan.org/testing/vlc-rtx-upscaler/vlc-3.0.19-nvidia-upscaler2.7z only the widescreen ratio that already fixed in the latest build.

There's something wrong with your log file, when the fullscreen is turned on, the log restarts as if it was a new VLC session:

direct3d11 debug: turning VSR ON
direct3d11 debug: picQuad position (0.00,0.00) 2562.00x1440.00
direct3d11 debug: DirectXUpdateRects source offset: 0,0 visible: 3440x1440
direct3d11 debug: DirectXUpdateRects image_src coords: 0,0,3440,1440
direct3d11 debug: DirectXUpdateRects image_src_clipped coords: 0,0,3440,1440
direct3d11 debug: DirectXUpdateRects image_dst coords: 0,0,3440,1440
direct3d11 debug: DirectXUpdateRects image_dst_clipped coords: 0,0,3440,1440
qt debug: Recente-- logger module started --
main debug: VLC media player - 3.0.19 Vetinari
main debug: Copyright © 1996-2022 the VideoLAN team
main debug: revision 3.0.18-158-g9c1c1edb45
main debug: configured with /builds/videolan/vlc/extras/package/win32/../../../configure  '--enable-update-check' '--enable-lua' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-avcodec' '--enable-merge-ffmpeg' '--enable-dca' '--enable-mpc' '--enable-libass' '--enable-schroedinger' '--enable-realrtsp' '--enable-live555' '--enable-shout' '--enable-goom' '--enable-sse' '--enable-mmx' '--enable-libcddb' '--enable-zvbi' '--disable-telx' '--enable-nls' '--host=x86_64-w64-mingw32' '--with-contrib=../contrib/x86_64-w64-mingw32' '--enable-debug' '--enable-qt' '--enable-skins2' '--enable-dvdread' '--enable-caca' 'host_alias=x86_64-w64-mingw32' 'CFLAGS= -D_WIN32_WINNT=0x0502 -DWINVER=0x502 -D__MSVCRT_VERSION__=0x700 ' 'CXXFLAGS= -D_WIN32_WINNT=0x0502 -DWINVER=0x502 -D__MSVCRT_VERSION__=0x700 ' 'PKG_CONFIG=pkg-config' 'PKG_CONFIG_LIBDIR=/usr/x86_64-w64-mingw32/lib/pkgconfig:/usr/lib/x86_64-w64-mingw32/pkgconfig'

Alright, here 2 new log files (without going fullscreen) : 1-VLC_RTX.log 2-VLC_RTX.log

Everything work correctly, but when I maximize the window or go on fullscreen there the issue begin and that too :

I see a few things:

the decoder restarts often with this pattern:

mp4 debug: elst (0) gives 382737ms (movie)-> 382778ms (track)
mp4 debug: track[Id 0x1] using Sync Sample Box (stss)
mp4 debug: stss gives 9177 --> 9108 (sample number)
mp4 debug: seeking with 2899ms preroll (use input-fast-seek to avoid)
mp4 debug: elst (0) gives 379837ms (movie)-> 379837ms (track)
main debug: Buffering 0%
mmdevice debug: state changed: 0
main debug: Buffering 7%
wasapi debug: reset
main debug: Buffering 13%
main debug: Buffering 19%
main debug: Buffering 26%
main debug: Buffering 32%
main debug: Buffering 38%
main debug: Buffering 45%
main debug: Buffering 51%
main debug: Buffering 58%
main debug: Buffering 64%
main debug: Buffering 70%
main debug: Buffering 77%
main debug: Buffering 83%
main debug: Buffering 89%
main debug: Buffering 96%
main debug: Stream buffering done (4028 ms in 4 ms)
main debug: end of audio preroll
wasapi debug: reset
main debug: Received first picture
main error: Timestamp conversion failed (delay 1000000, buffering 100000, bound 9000000)
main error: Could not convert timestamp 49253607868 for FFmpeg
avcodec debug: available hardware decoder output format 53 (dxva2_vld)
avcodec debug: available hardware decoder output format 118 (d3d11va_vld)
avcodec debug: available hardware decoder output format 174 (d3d11)
avcodec debug: available software decoder output format 0 (yuv420p)
avcodec debug: reusing decoder output format 118

lots of pictures are dropped in VSR mode (although your GPU should be plenty fast enough)

picture is too late to be displayed (missing 33 ms)

it ends up dropping data which might be where the leaks happen

buffer too late (-736930 us): dropped

you're using 3158 drivers while I'm on 3141

Can you share your file so I can check if I get the same behaviour here ? My test MP4 with bigger size don't have this kind of issue.

Also are you playing from a slow disk/USB drive ? Because all the buffering is not normal.

Also you mentioned that there's no problem with the special RTX build of VLC. Can you provide the logs for the same file with that build ?

I found a way to reproduce the issue:

resize the window manually a lot
breakpoint in the code to simulate slow input
start again resizing until it starts leaking

This does not happen with the VideoProcessor output which uses the same code path as the Super Resolution output. It might be a problem in the driver when the GPU load is too much (I get some device disconnection when resizing a lot, the GPU being using close to 50% of resources).

This is also happens on the 4.0 branch.

I have an even simpler way to trigger the issue:

launch playback
maximize the window

The GPU starts leaking memory. At some point it maxes out the memory usage and starts leaking CPU memory (!).

If you go into fullscreen instead of maximizing the window, it doesn't leak. On our side we call exactly the same thing between the 2 behaviours. Only the dimensions are slightly different between the two.

There's also no leak when maximizing in VideoProcessor mode. The only difference being this code to enable the NVIDIA super resolution: https://code.videolan.org/videolan/vlc/-/blob/c585c486dea97c169c56eef9ef674887ec368ee7/modules/video_output/win32/d3d11_scaler.cpp#L261

When the leak happens there's also a crash in nvwgf2umx.dll when releasing the ID3D11Device. It is the last D3D11 object we hold at that point. Everything else being NVIDIA's internal data.

lots of pictures are dropped in VSR mode
it ends up dropping data which might be where the leaks happen

About those, it happens after the ram goes to 100% and break the video images, not before

you're using 3158 drivers while I'm on 3141 :

yeah, but I retried today with 3179 drivers and 09.05.23 nightly build, the same file and advanced the video towards the end, when the memory becoming to increase more than 5GB : Log : 3-VLC_RTX.log Video : https://we.tl/t-Ns1qmhRDeZ

It happens with all type of video files, the main video file that use to test is on SSD, but I already tried with another from a USB drive or Hard Drive: avi, mkv. but it remains the same.

Mostly it happens when the window are bigger, just one time not a lot, in my side and when the VLC memory usage is superior to 250 MB, then it start to double untill 100% of usage and then dropped for the vlc.exe process back to approximately 90 MB without clearing memory and then break the video because there's no free memory to use, but nothing happen the sound it's perfect even the video breaks.

I forgot to say that it happen on Nvidia SuperResolution level 3 and 4, didn't try with 1 and 2, and I'm using a widescreen.

Here the special RTX Vetinari Build log without memory leak in maximize or fullscreen only the ratio widescreen issue : VLC_Special_RTX_Build.log

I'm gonna try with VideoProcessor Way, with both build special RTX and latest nightly.

With the nightly build set to VideoProcessor there's no memory leak or memory biggest increase. But I don't see any difference in the quality or any kind of upscaling, not like SuperResolution setting, here the log :4-VLC_RTX.log

In other side with the special RTX build when d3d11 is set to VideoProcessor there's an upscaling and quality increase : VLC_Special_RTX_Build-2.log

The working branch is https://code.videolan.org/robUx4/vlc/-/tree/30-super-res/7, now I need to find out why it's working there...

I found the culprit: it's the cropping/stretching of the original texture here: https://code.videolan.org/videolan/vlc/-/blob/7cc1c33b703fc0ea12babac6e264245e83556902/modules/video_output/win32/d3d11_scaler.cpp#L242

The values seem legit though, they do produce the proper result and give no issue with the basic VideoProcessor. That sounds like an issue with the NVIDIA drivers that fills the buffer in the right place but leaks when doing that...

hopefully this issue will be fixed

We won't release without a fix, or a driver update that fixes the issue.

And by the way, thanks for the bug report on a nightly build and the help figuring this out

changed milestone to %3.0.19

mentioned in merge request !3664 (closed)

I can no longer reproduce the issue with NVIDIA drivers 531.79 and Win10 19045.3208. According to NVIDIA they fully fixed it with drivers 535.98 from 2023/05/30.

closed

added Status::fixed label

VLC Super Resolution Memory Leak

Child items ...

Activity