avcodec: Double free while transcoding with FFmpeg encoder
VLC master branch crashes with double free while using lavc encoder (i.e. h264_amf) for video transcoding.
To reproduce the crash, you need to make the following actions:
- Choose
h264_amf
encoder in encoding section (Tools -> Preferences -> Show settings (All) -> Input/Codecs -> Video Codecs -> FFmpeg). - To force using lavc encoders for transcoding, you need to choose
FFmpeg audio/video encoder
in the menu: Tools -> Preferences -> Stream Output -> Sout stream -> Transcode section. - Choose some video file and convert to h264 video format thru Media -> Convert / Save menu
This issue doesn't occur on the 3.0.18 branch.
The crash occures at the end of transcoding due to double calling of EndVideoEnc function in
transcode_video_clean function.
In fact, transcode_encoder_close and transcode_encoder_delete are both calling encoder->ops->close
for lavc encoder, but I'm not sure that the second close in vlc_encoder_Destroy is really needed.
Stack trace:
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7d81859 in __GI_abort () at abort.c:79
#2 0x00007ffff7dec26e in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7f16298 "%s\n")
at ../sysdeps/posix/libc_fatal.c:155
#3 0x00007ffff7df42fc in malloc_printerr (str=str@entry=0x7ffff7f18690 "double free or corruption (!prev)") at malloc.c:5347
#4 0x00007ffff7df5fac in _int_free (av=0x7fff3c000020, p=0x7fff3c1d99d0, have_lock=<optimized out>) at malloc.c:4317
#5 0x00007fff49a1d16b in EndVideoEnc (p_enc=<optimized out>) at ../../modules/codec/avcodec/encoder.c:1503
#6 0x00007ffff7c8e0a7 in vlc_encoder_Destroy (encoder=0x7fff3c1d8610) at ../../src/input/decoder_helpers.c:176
#7 0x00007fff855554d7 in transcode_encoder_delete (p_enc=0x7fff3c0e06a0) at ../../modules/stream_out/transcode/encoder/encoder.c:55
#8 0x00007fff85558bfe in transcode_video_clean (id=id@entry=0x7fff44c1bc70) at ../../modules/stream_out/transcode/video.c:403
#9 0x00007fff85553236 in Del (p_stream=0x7fff44004860, _id=0x7fff44c1bc70) at ../../modules/stream_out/transcode/transcode.c:710
#10 0x00007ffff7cfa11f in sout_StreamIdDel (s=s@entry=0x7fff44004860, id=0x7fff44c1bc70) at ../../src/stream_output/stream_output.c:707
#11 0x00007ffff7cfa17c in sout_InputDelete (p_sout=0x7fff44004860, p_input=0x7fff44c07ef0)
at ../../src/stream_output/stream_output.c:155
#12 0x00007ffff7c8ab22 in DeleteDecoder (p_owner=p_owner@entry=0x7fff44c0ef10, i_cat=VIDEO_ES) at ../../src/input/decoder.c:2054
#13 0x00007ffff7c8ce70 in vlc_input_decoder_Delete (p_owner=0x7fff44c0ef10) at ../../src/input/decoder.c:2287
#14 0x00007ffff7c91537 in EsOutDestroyDecoder (out=out@entry=0x7fffd40f4160, p_es=p_es@entry=0x7fff44c0d440)
at ../../src/input/es_out.c:2412
#15 0x00007ffff7c92805 in EsOutUnselectEs (out=out@entry=0x7fffd40f4160, es=es@entry=0x7fff44c0d440, b_update=<optimized out>)
at ../../src/input/es_out.c:2581
#16 0x00007ffff7c97afe in EsOutVaPrivControlLocked
(out=out@entry=0x7fffd40f4160, query=query@entry=65536, args=args@entry=0x7fff84269b60) at ../../src/input/es_out.c:3745
#17 0x00007ffff7c988bf in EsOutPrivControl (out=0x7fffd40f4160, query=65536, args=0x7fff84269b60) at ../../src/input/es_out.c:4040
#18 0x00007ffff7c9c009 in es_out_vaPrivControl (args=0x7fff84269b60, query=<optimized out>, out=<optimized out>)
at ../../src/input/es_out.h:105
#19 es_out_PrivControl (out=<optimized out>, query=<optimized out>) at ../../src/input/es_out.h:112
#20 0x00007ffff7c9c0b3 in CmdExecutePrivControl (p_tsout=p_tsout@entry=0x7fff440034f8, p_cmd=p_cmd@entry=0x7fff84269c50)
at ../../src/input/es_out_timeshift.c:1809
#21 0x00007ffff7c9c5e1 in PrivControlLocked
(p_tsout=p_tsout@entry=0x7fff440034f8, i_query=i_query@entry=65536, args=args@entry=0x7fff84269ce0)
at ../../src/input/es_out_timeshift.c:799
#22 0x00007ffff7c9c7e7 in PrivControl (p_tsout=0x7fff440034f8, i_query=65536, args=0x7fff84269ce0)
at ../../src/input/es_out_timeshift.c:858
#23 0x00007ffff7c9de46 in es_out_vaPrivControl (args=0x7fff84269ce0, query=65536, out=<optimized out>) at ../../src/input/es_out.h:105
#24 es_out_PrivControl (out=<optimized out>, query=query@entry=65536) at ../../src/input/es_out.h:112
#25 0x00007ffff7ca3fce in es_out_SetMode (i_mode=0, p_out=<optimized out>) at ../../src/input/es_out.h:119
#26 End (p_input=p_input@entry=0x7fffd59e5e00) at ../../src/input/input.c:1400
#27 0x00007ffff7ca4355 in Run (data=0x7fffd59e5e00) at ../../src/input/input.c:431
#28 0x00007ffff7f59609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#29 0x00007ffff7e7e133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95