VLC segfaults playing H.264 streamed from a RTSP server with hw acceleration

The segmentation fault occurs when the machine is lagging behind when playing a RTSP stream, e.g. because of network issues. After less than one minute it would crash. Our streams are particular in a way, as buffer dimensions keep jumping from 720x578 to 720x576.

On setups where the network is ok, the same stream can be played without crashes. Occurs when dxva2 hardware acceleration is enabled.

Stack trace:

Program received signal SIGSEGV, Segmentation fault.

[#0](https://code.videolan.org/videolan/vlc/-/issues/0)  0x536c1d92 in ReleasePictureSys (p_sys=0x24888454) at ../../extras/package/win32/../../../modules/codec/avcodec/../../video_chroma/d3d9_fmt.h:84
[#1](https://code.videolan.org/videolan/vlc/-/issues/1)  d3d9_pic_context_destroy (opaque=0x24888448) at ../../extras/package/win32/../../../modules/codec/avcodec/dxva2.c:174
[#2](https://code.videolan.org/videolan/vlc/-/issues/2)  0x5fcafacb in PictureDestroyContext (p_picture=0x247e5cf8) at ../../extras/package/win32/../../../src/misc/picture.c:100
[#3](https://code.videolan.org/videolan/vlc/-/issues/3)  picture_Release (p_picture=0x247e5cf8) at ../../extras/package/win32/../../../src/misc/picture.c:314
[#4](https://code.videolan.org/videolan/vlc/-/issues/4)  0x69f8ec73 in buffer_replace (src=0x0, dst=0x2674e298) at libavutil/buffer.c:120
[#5](https://code.videolan.org/videolan/vlc/-/issues/5)  av_buffer_unref (buf=buf@entry=0x2674e298) at libavutil/buffer.c:130
[#6](https://code.videolan.org/videolan/vlc/-/issues/6)  0x69fa406b in av_frame_unref (frame=frame@entry=0x2674e1a0) at libavutil/frame.c:520
[#7](https://code.videolan.org/videolan/vlc/-/issues/7)  0x69fa5256 in av_frame_unref (frame=0x2674e1a0) at libavutil/frame.c:535
[#8](https://code.videolan.org/videolan/vlc/-/issues/8)  0x6971f2da in ff_thread_release_buffer (avctx=<optimized out>, f=f@entry=0x2477b54c) at libavcodec/pthread_frame.c:998
[#9](https://code.videolan.org/videolan/vlc/-/issues/9)  0x69b5bdca in ff_h264_unref_picture (h=h@entry=0x2477b000, pic=pic@entry=0x2477b548) at libavcodec/h264_picture.c:53
[#10](https://code.videolan.org/videolan/vlc/-/issues/10) 0x69b664ec in ff_h264_update_thread_context (dst=<optimized out>, src=<optimized out>) at libavcodec/h264_slice.c:386
[#11](https://code.videolan.org/videolan/vlc/-/issues/11) 0x6971c438 in update_context_from_thread (dst=0x24acd200, src=0x24de0700, for_user=for_user@entry=0) at libavcodec/pthread_frame.c:310
[#12](https://code.videolan.org/videolan/vlc/-/issues/12) 0x6971cbad in submit_packet (avpkt=<optimized out>, user_avctx=<optimized out>, p=<optimized out>) at libavcodec/pthread_frame.c:413
[#13](https://code.videolan.org/videolan/vlc/-/issues/13) ff_thread_decode_frame (avctx=<optimized out>, avctx@entry=0x1c7ea440, picture=<optimized out>, picture@entry=0x25fe6b80,
    got_picture_ptr=<optimized out>, got_picture_ptr@entry=0x1fcffc88, avpkt=<optimized out>, avpkt@entry=0x1c9167e0)
    at libavcodec/pthread_frame.c:490
[#14](https://code.videolan.org/videolan/vlc/-/issues/14) 0x69487c60 in decode_simple_internal (frame=0x25fe6b80, avctx=0x1c7ea440) at libavcodec/decode.c:396
[#15](https://code.videolan.org/videolan/vlc/-/issues/15) decode_simple_receive_frame (frame=<optimized out>, avctx=<optimized out>) at libavcodec/decode.c:594
[#16](https://code.videolan.org/videolan/vlc/-/issues/16) decode_receive_frame_internal (avctx=avctx@entry=0x1c7ea440, frame=0x25fe6b80) at libavcodec/decode.c:612
[#17](https://code.videolan.org/videolan/vlc/-/issues/17) 0x69488947 in avcodec_send_packet (avctx=avctx@entry=0x1c7ea440, avpkt=avpkt@entry=0x1fcffd90) at libavcodec/decode.c:674
[#18](https://code.videolan.org/videolan/vlc/-/issues/18) 0x69252fdb in DecodeBlock (error=<synthetic pointer>, pp_block=<optimized out>, p_dec=<optimized out>)
    at ../../extras/package/win32/../../../modules/codec/avcodec/video.c:1022
[#19](https://code.videolan.org/videolan/vlc/-/issues/19) DecodeVideo (p_dec=<optimized out>, p_block=<optimized out>) at ../../extras/package/win32/../../../modules/codec/avcodec/video.c:1234
[#20](https://code.videolan.org/videolan/vlc/-/issues/20) 0x5fc5d4b8 in DecoderDecode (p_dec=0x1c90b988, p_block=0x26d668c0) at ../../extras/package/win32/../../../src/input/decoder.c:1332
[#21](https://code.videolan.org/videolan/vlc/-/issues/21) 0x5fc5d2b9 in DecoderProcess (p_dec=p_dec@entry=0x1c90b988, p_block=p_block@entry=0x249eb378)
    at ../../extras/package/win32/../../../src/input/decoder.c:1440
[#22](https://code.videolan.org/videolan/vlc/-/issues/22) 0x5fc5d92f in DecoderThread (p_data=<optimized out>) at ../../extras/package/win32/../../../src/input/decoder.c:1611
[#23](https://code.videolan.org/videolan/vlc/-/issues/23) 0x5fcc2b1d in vlc_entry (p=0x1c930d60) at ../../extras/package/win32/../../../src/win32/thread.c:470
[#24](https://code.videolan.org/videolan/vlc/-/issues/24) 0x77781287 in msvcrt!_itow_s () from C:\Windows\syswow64\msvcrt.dll
[#25](https://code.videolan.org/videolan/vlc/-/issues/25) 0x77781328 in msvcrt!_endthreadex () from C:\Windows\syswow64\msvcrt.dll
[#26](https://code.videolan.org/videolan/vlc/-/issues/26) 0x7633336a in KERNEL32!BaseThreadInitThunk () from C:\Windows\syswow64\kernel32.dll
[#27](https://code.videolan.org/videolan/vlc/-/issues/27) 0x77c598f2 in ntdll!RtlInitializeExceptionChain () from C:\Windows\SysWOW64\ntdll.dll
[#28](https://code.videolan.org/videolan/vlc/-/issues/28) 0x77c598c5 in ntdll!RtlInitializeExceptionChain () from C:\Windows\SysWOW64\ntdll.dll
[#29](https://code.videolan.org/videolan/vlc/-/issues/29) 0x00000000 in ?? ()

Platform: Intel Core i7-6700 NVidia Quadro M5000 Windows 7 Ultimate Version 6.1 (Build 7601: Service Pack 1) Reproduced with vlc-3.0.1-20180301-0224 Debug version

Added attachment:

• log.txt

assigned to @robUx4

changed the description

added Component::Video: Direct3D / Win32 label

Please try with 3.0.2- from today.

Your log file is weird. There's twice the same error from the DXVA2 error as if it was executed simultaneously which should be impossible. Can you produce a log file (verbosity 2) without editing it, for example with a redirection on the command line (since it crashes you won't be able to use Save As from the Message dialog).

Log when trying to display vertical moving bars

Added attachment:

• bars_video.txt

I tried with 3.0.2 and still crahes. I have added a new log file, this time playing the video without hardware acceleration.

But is this crashing without the hardware acceleration ? If so can you provide the backtrace because it shouldn't be the same as the one you provided before.

Log when using dxva2 hardware acceleration

Added attachment:

• bars_video_with_dxva.txt

It does not crash without hardware acceleration. I added now the log using dxva2.

changed title from VLC segfaults playing H.264 streamed from a RTSP server to VLC segfaults playing H.264 streamed from a RTSP server with hw acceleration

changed milestone to %3.0.x maintenance

Fixed it. It was a buffer overrun when using DVXA2 and D3D11.

By the way, when using D3D11 (default on Windows 10) you should use D3D11VA not DXVA2 otherwise you're losing a lot of performance.

changed milestone to %3.0.2

Fixed by ||= cdbd28fa78 =|| chroma: copy: fix buffer overrun when the destination pitch is smaller than the src ||

Backported as f18f5e6c0611165248bff15281ad33318599c5ba

added Status::fixed label

closed

With vlc-3.0.2-20180322-0549, it crashes less often, but still crashes. I have not been able to get a new stacktrace, cause gdb 4.5 (which I used to generate the previous stack trace) says that this new vlc.exe is not executable format.

(I was using the debug version of course)

Without stracktrace, it's hard to debug...

For the record, I did not manage to reproduce that particular issue but something similar enough. Apparently that wasn't enough. Do you have a capture of the stream ? Maybe record it to TS so it's similar enough to the source ?