Skip to content

transcode: rework `encoder_close` to fix remaining double-frees

Alaric Senat requested to merge asenat/vlc:transcode-enc-close into master

Single module_unneed calls aren't enough to properly close encoders as their disable callbacks are stored independently from the module.
As a side effect, this MR fixes a double-free where the custom close callback was called after the module_unneed call in the encoder destroy function:

READ of size 8 at 0x619000050498 thread T10 (vlc-input)
    #0 0x7f83c0e72a67 in CloseEncoder ../../modules/codec/x264.c:1523
    #1 0x7f83d64b7613 in vlc_encoder_Destroy ../../src/input/decoder_helpers.c:176
    #2 0x7f83ccb31986 in transcode_encoder_delete ../../modules/stream_out/transcode/encoder/encoder.c:55
    #3 0x7f83ccb53203 in transcode_video_clean ../../modules/stream_out/transcode/video.c:405
    #4 0x7f83ccb2f6ac in Del ../../modules/stream_out/transcode/transcode.c:736
    #5 0x7f83d679814f in sout_StreamIdDel ../../src/stream_output/stream_output.c:707
    #6 0x7f83d6791cc3 in sout_InputDelete ../../src/stream_output/stream_output.c:155
    #7 ...

0x619000050498 is located 1048 bytes inside of 1072-byte region [0x619000050080,0x6190000504b0)
freed by thread T10 (vlc-input) here:
    #0 0x7f83d73b6388 in __interceptor_free.part.0 (/lib64/libasan.so.8+0xb9388)
    #1 0x7f83d6740fcd in vlc_objres_clear ../../src/misc/objres.c:93
    #2 0x7f83d640265d in module_unneed ../../src/modules/modules.c:307
    #3 0x7f83ccb3f212 in transcode_encoder_video_close ../../modules/stream_out/transcode/encoder/video.c:390
    #4 0x7f83ccb331a2 in transcode_encoder_close ../../modules/stream_out/transcode/encoder/encoder.c:188
    #5 0x7f83ccb53178 in transcode_video_clean ../../modules/stream_out/transcode/video.c:404
    #6 0x7f83ccb2f6ac in Del ../../modules/stream_out/transcode/transcode.c:736
    #7 ...
Edited by Alaric Senat

Merge request reports