Skip to content

mp4: essetup: fix QTRLE decoding

Alexandre Janniaux requested to merge alexandre-janniaux/vlc:qtrle-fix-regression/1 into master

Video using the ffmpeg qtrle decoder are failing to play with:

[qtrle @ 0x76117804fc00] Unsupported colorspace: 0 bits/sample?
[0000761178024760] avcodec decoder error: cannot start codec (qtrle)
[0000761178024760] main decoder error: Codec `rle ' (Apple QuickTime RLE Video) is not supported.
[0000761178024760] main decoder error: Codec not supported
[0000761178024760] main decoder error: VLC could not decode the format "rle " (Apple QuickTime RLE Video)

Indeed, QTRLE format can compress source with either 1, 2, 4, 8, 16, 24 or 32 bpp, and the actual bpp is stored in the stst atom, not in the decoder payload. The demux has to forward this value to the decoder one way or another.

Regression from 12030f5ccac852aaa3f9f66b20606d2fc344f8f4, which starts setting the bpp to zero in avcodec because there is no chroma yet. Defining a chroma instead of forwarding the bpp would be possible for 8, 16, 24 RLE data cases, but would not work for lower RLE data cases.

The correct i_bits_per_pixel field being removed in commit 1d46d8b7, a hack is implemented instead. The hack is from 3e5f2e5d and other places that needed it.

Fixes #28550 (closed)

Merge request reports

VideoLAN code repository instance