chromecast spu subtitles support

** Direct mux of subtitles into media ?

WebVTT seems the only option if we want to use mkv.

• WebVTT can only be muxed (using old deprecated mapping) into Webm by ffmpeg https://trac.ffmpeg.org/ticket/5434
• Webm ffmpeg mux makes VP8/9 codec and vorbis mandatory

Chromecast does not seem to "see" the embedded webvtt track. Maybe it needs to "enable" that track using a dedicated command.

"Your subtitle resources must implement CORS." https://developers.google.com/cast/docs/media So it seems all subtitles are split file.

** Side loaded subtitles ?

• We need the "command"
• There's no guarantee the chromecast does not expects a complete file when accessing it. (and the way WebVTT uses timestamps tends to make this mandatory)

We would need to extract/transcode all subtitles first.

The only option in that case seems to provide subtitles in segments (HLS, ...), which probably the whole media to be adaptive.

** Burn in subtitles ?

• Makes transcoding mandatory

added Component::Stream output Severity::normal Type::feature Version::master git + 1 deleted label

changed milestone to %Features paradize

changed the description

https://github.com/googlecast/CastReferencePlayer/blob/master/player.js#L1581

https://github.com/thibauts/node-castv2-client/wiki/How-to-use-subtitles-with-the-DefaultMediaReceiver-app

Declaring Media

+    ss << "\"contentId\":\"" << chromecast_url.str() << "\""
+       << ",\"streamType\":\"BUFFERED\""
+       << ",\"contentType\":\"video/webm\""
+       << ",\"tracks\": [{"
+       <<   "\"trackId\": 3,"
+       <<   "\"trackContentType\": \"text/vtt\","
+       <<   "\"type\": \"TEXT\","
+       <<   "\"subType\": \"SUBTITLES\","
+       <<   "\"language\": \"en-US\""              /* mandatory */
+       <<  "}]";

Sending extra activation

+    ss << "{\"type\":\"EDIT_TRACKS_INFO\","
+       <<  "\"activeTrackIds\":[3],"
+       <<  "\"enableTextTracks\":\"true\","
+       <<  "\"mediaSessionId\":" << mediaSessionId << ","
+       <<  "\"requestId\":" << m_requestId++
+       << "}";

Or requesting activation from Load command

     ss << "{\"type\":\"LOAD\","
        <<  "\"media\":{" << GetMedia( i_port, title, artwork, mime ) << "},"
        <<  "\"autoplay\":\"false\","
+       <<  "\"enableTextTracks\":\"true\","
+       <<  "\"activeTrackIds\":[3],"
        <<  "\"requestId\":" << m_requestId++
        << "}";

won't change a thing for webvtt in webm

See #19648 (closed) and #19958 (closed)

added Component::Stream output: Chromecast label and removed Component::Stream output label

Seems like it supports embedded subtitles if it is muxed as mp4:

https://github.com/muammar/mkchromecast/blob/8b52a42377deebc3c88ef1b3264e542b8fe49277/mkchromecast/video.py#L136

vs

https://github.com/muammar/mkchromecast/blob/8b52a42377deebc3c88ef1b3264e542b8fe49277/mkchromecast/video.py#L230

is there any specific reason for using mkv?

Looks like Airflow (https://airflow.app/) (proprietary) can do subtitles with both mp4 and mkv muxed h264 without transcoding. Basic packet capture shows it uses GStreamer.

changed title from chromecast spu support to chromecast spu subtitles support

This person added subtitles casting to chromecast using VLC:

https://gitlab.com/nirhal/vlc-cc-spu/-/commits/cc-spu-fix

Seems like it will be implemented in version 4 of vlc: vlc-android#502 (comment 45323)

mentioned in issue #19648 (closed)

mentioned in issue #19958 (closed)

mentioned in issue #20292 (closed)

mentioned in issue #21396 (closed)

mentioned in issue #23617 (closed)

mentioned in issue #24615 (closed)

mentioned in issue #24712 (closed)

mentioned in issue #25027 (closed)

marked #25079 (closed) as a duplicate of this issue

marked this issue as related to #25079 (closed)

marked #19958 (closed) as a duplicate of this issue

marked this issue as related to #19958 (closed)

removed Severity::normal label

removed Version::master git label

marked #25632 (closed) as a duplicate of this issue

marked this issue as related to #25632 (closed)

marked #25695 (closed) as a duplicate of this issue

marked this issue as related to #25695 (closed)

marked #26249 (closed) as a duplicate of this issue

marked this issue as related to #26249 (closed)

Is this still slated for VLC4? Would be really cool to see this finally released :)

Any nightly builds I could try perhaps? Is it already in master?

We are still working on it for VLC 4.0.

Hey, how is the implementation going to work for image based subtitles? Airflow uses OCR. @tguillem

Also, what does SPU stand for?

SPU stands for subpicture unit http://www.mpucoder.com/guides/subpic3.html

The plan is to blend the subtitle image into the video via an GL offscreen surface (a slight CPU/GPU extra cost then), but OCR is also a good idea.

marked vlc-android#502 (closed) as a duplicate of this issue

marked this issue as related to vlc-android#502 (closed)

marked #27021 (closed) as a duplicate of this issue

marked this issue as related to #27021 (closed)

unassigned @MigrationBot

assigned to @asenat

Any update on this?

The chromecast module is being rewritten to expose the media via HLS, which solves many problems we have (including proper subtitle support). This feature is still planned for 4.0 :)

marked this issue as related to #27375 (closed)

changed milestone to %4.0