[WIP] Thumbnails Previewing Support

Previews Thumbnails

This is documentation for the WIP regarding the media preview thumbnails in the VLC interface

Existing specifications

DASH-IF

DASH has already specified thumbnailing as a tiled and segmented format.

6.2.6. of https://dashif.org/docs/DASH-IF-IOP-v4.1-clean.pdf

• Thumbnails are tiled into one or more compressed images for efficiency
• Each image is served as a Segment
• Each segment has its duration
• Each Image has a COLS*ROWS resolution
• The pixel resolution of the tiles is inferred from the image sample dimensions
• The tile/thumbnail duration is inferred from the sample/segment duration and ROWS*COLS
• Since this is seen as another AdaptationSet, it supports resolution variants

Example manifest: http://dash.akamaized.net/akamai/bbb_30fps/bbb_with_multiple_tiled_thumbnails.mpd

<AdaptationSet id="3" mimeType="image/jpeg" contentType="image">
<SegmentTemplate media="$RepresentationID$/tile_$Number$.jpg" duration="634.566" startNumber="1"/>
<Representation bandwidth="12000" id="thumbnails_102x58" width="1024" height="1152">
<EssentialProperty schemeIdUri="http://dashif.org/thumbnail_tile" value="10x20"/>
</Representation>
<Representation bandwidth="24000" id="thumbnails_256x144" width="2048" height="1152">
<EssentialProperty schemeIdUri="http://dashif.org/thumbnail_tile" value="8x8"/>
</Representation>
</AdaptationSet>

Which results as a single image for the whole period: https://dash.akamaized.net/akamai/bbb_30fps/thumbnails_102x58/tile_1.jpg

HLS

Does not appear in the IETF specification but referred by many packagers resources:

EXT-X-TILES

The EXT-X-TILES tag indicates the Image Resource is an Image Archive that contains one or more Images arranged in a Grid. A client processing the EXT-X-TILES tag MUST be capable of displaying individual Image Tiles using the Grid Sequence and Timing Model.

https://developer.roku.com/docs/developer-program/media-playback/trick-mode/hls-and-dash.md

https://github.com/image-media-playlist/spec/blob/master/image_media_playlist_v0_4.pdf

• Served in a similar way of DASH
• It declares both pixel resolution and thumbnail density (rows/cols)
• Each tile has also an explicit duration

https://raw.githubusercontent.com/dvoracek-slub/shaka-player-issue/main/hls/thumbnails.m3u8

#EXTM3U
#EXT-X-TARGETDURATION:400
#EXT-X-VERSION:7
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-IMAGES-ONLY
#EXTINF:400,
#EXT-X-TILES:RESOLUTION=80x45,LAYOUT=10x10,DURATION=4
tile_1.jpg
#EXT-X-ENDLIST

My Initial POC

I made few POC tries to render the DASH content.

It is abusing the ATTACHMENT API by defining a custom name: "thumbnail://$version/$wx$h/$start-$duration"

https://code.videolan.org/fcartegnie/vlc/-/tree/tiledthumbnails/1?ref_type=heads

Enregistrement d'écran_20250718_151507.webm

Outcome is:

• I thought I could simplify using ATTACHMENT instead of a vlc_previews/GET_PREVIEWS APIs, but I later realized this no longer was able to have demux manage the thumbnails lifeycle (with Live content it needs to expire and continuously insert new ones) and possible variants.
• I have spent lots of time trying to use the QML native tiling (sourceClipRect). It worked with a static image in QRC, but in the end, it is impossible to perform the tiling when the image isn't coming from the Qt resources provider. As our content comes from memory I had to switch from a custom image provider for the POC. Fatih has an idea about performing the view using textures.

On the fly POC from contributor

@mvbrahmam33 proposed recently its own implementation, generating the thumbnails during playback and though Qt. While the architecture can not fit, he also has its own implementation of thumbnails display in the UI.

MR for reference: !7822 (closed)

Thumbnails generation

@gabriel_lt worked on medialibrary to consider generating thumbnails according to my own POC nomenclature.

It appears it raises many questions:

• Thumbnailing frequency
• How to store thumbnails
• How to pass thumbnails to the core (as metadata ? custom API ?)
• How to request thumnailing

Medialibrary and discussions

VideoLabs had an internal meeting end September regarding medialibrary. This meeting also addressed the previews thumbnails generation.

Notes: medialibrary#493 (closed)

Considerations

• Thumbnails could come from multiple sources, just like subtitles:
  • Extracted from the media by the demuxer (ex: DASH/HLS)
  • Sideloaded along the media file (just like SRT subtitles).
  • Generated on request, then,
  • Provided by medialibrary

Core modifications

• Add API struct vlc_previews
• Add input controls to fetch PREVIEWS
• vlc_player/mediaplayer API to side load
• Manage lifecycle

Other considerations

Central database service ?

It could be interesting to have a repository for thumbnails previews, just like opensubtiles for subtitles. This would prevent generating locally the content.

Thumbnails as Sideloaded files

Shipping the thumnails as sidecar file just like subtitles would ease adoption.

We will need to define a filename nomenclature and push for its adoption. I have few ideas.

Media storage

Storing the thumbnails tiled or separate within the media could be also possible. (Samples ? Dedicated metadata properties ?) It will also require specification process.

A dirty hack (ID3 headers ) could be also used with dinosaurs formats.