Missing ass_set_storage_size can lead to distorted ASS subtitles
VLC does not tell libass the storage size of the video; in that case libass guesses the storage size — but that guess isn't and cannot be always correct. If the correct storage size was not passed and the guess doesn't describe reality certain ASS tags, like e.g. 3D-rotations, will be distored.
ass_set_storage requires “the actual storage size of the video stream, without any anamorphic de-squeeze applied” to be passed (see libass docs).
I attached sample files and subs affected by this issue (one anamorphic and one non-anamorphic): Samples: vlc-issue_libass-storage-size.tar.xz
To get a correct rendering for comparison e.g. mpv, xy-VSFilter/xy-SubFilter or classic MPC may be used.
For the 1080p sample, VLC renders the following:
but the correct rendering is this:
.
For the anamorphic sample, VLC renders
but should instead render
This was tested with recent libass and VLC 3.0.16 (from Debian Bullseye, package version 3.0.16-1) but since there's no call to ass_set_storage_size in current git master (54088315) I highly doubt the main problem has changed since.
On a side note, unrelated to the missing storage size, I noticed that at least with the tested version the snapshots taken from VLC’s menu do
- for the non-anamorphic 1080p sample file change the colour compared to what’s shown during playback. This does not happen for screenshots taken from X11 as used above or snapshots taken with mpv for the same file.
- for the anamorphic sample-file the snapshot has storage size resolution instead of display size resolution resulting in the snapshot being squeezed along one dimension. I guess this is not intentional?
The snapshots taken by VLC are included in the attached archive.