Implement IPT-PQ and reshaper

@haasn

including brute-force solving for the matrix based on a known input and output!

Oh, wow. That is a bad idea, but you got the second matrix (PL_COLOR_SYSTEM_DOLBY_IPT) almost correct. Compare this and [0050] from patent. Actually it should be this one. Again without the reshaping, it should not be this one.

Now second matrix dvipt_lms2rgb is this one (calculations to get it attached in pic) (then round to 4th decimal, invert and then transpose):

BT.2111 test pattern that was generated by testing-av project is wrong IMHO. But you can use it to check what it should be because it does decode in Films & TV and that Dolby vision app.

changed the description

I'm not sure I follow. You mention a lot of matrices. Can you summarize which matrix you believe is the correct YCbCr->LMS matrix that we should hardcode?

Edit: Nvm, your first two links were just demonstrating why the existing matrix doesn't match up.

So, I implemented your suggestions, which upon inspection I realized were pretty much the first thing I tried as well. The problem is, suffice to say, that it does not work, and the output is not useful. I've pushed the commit at 3c19e385. Attached are screenshots demonstrating the issue.

Using the following dolby vision sample as a source:

λ mediainfo iOS_P5_GlassBlowing2_3840x2160@59.94fps_15200kbps.mp4
General
Complete name                            : iOS_P5_GlassBlowing2_3840x2160@59.94fps_15200kbps.mp4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (mp42/isom/iso5)
File size                                : 330 MiB
Duration                                 : 2 min 58 s
Overall bit rate                         : 15.6 Mb/s

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@Main
HDR format                               : Dolby Vision, Version 1.0, dvhe.05.09, BL+RPU
Codec ID                                 : dvh1
Duration                                 : 2 min 55 s
Bit rate                                 : 15.2 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 59.940 (60000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 0.031
Stream size                              : 317 MiB (96%)
Default                                  : Yes
Alternate group                          : 1
Color range                              : Full
Codec configuration box                  : hvcC+dvcC

I get the following screenshot:

Compare this against what the file is supposed to look like (BT.2020 source):

λ mediainfo P81_GlassBlowing2_3840x2160@59.94fps_15200kbps_fmp4.mp4
General
Complete name                            : P81_GlassBlowing2_3840x2160@59.94fps_15200kbps_fmp4.mp4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (mp42/isom/iso5)
File size                                : 329 MiB
Duration                                 : 2 min 58 s
Overall bit rate                         : 15.5 Mb/s

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@Main
HDR format                               : Dolby Vision, Version 1.0, dvhe.08.09, BL+RPU, HDR10 compatible / SMPTE ST 2086, HDR10 compatible
Codec ID                                 : hev1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 2 min 55 s
Bit rate                                 : 15.1 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 59.940 (60000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 2)
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.030
Stream size                              : 316 MiB (96%)
Default                                  : Yes
Alternate group                          : 1
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : BT.2020
Mastering display luminance              : min: 0.0050 cd/m2, max: 4000 cd/m2
Codec configuration box                  : hvcC+dvvC

This produces the much more reasonable-looking output:

Do you see now why I believe Profile 5 output is completely useless/meaningless without reshaping support?

added enhancement help wanted wishlist labels

Interestingly enough yellow is more accurate on first picture, at least that is how reference Dolby decoder plays IPT-PQ file in windows 10.

All tests IMHO should be done on a very simple file like this one https://github.com/sekrit-twc/zimg/files/3894102/Dolby_Vision_Test_Pattern_Demo.zip and not include reshaping change...

BTW, I totally forgot, profile 5 is (always) Full range. Do we remember that? I am asking because there are some almost black dots (0, 6, 4; e.g.) in first picture that are (47, 45, 32) in second picture.

BTW, I totally forgot, profile 5 is (always) Full range. Do we remember that? I am asking because there are some almost black dots (0, 6, 4; e.g.) in first picture that are (47, 45, 32) in second picture.

No, I forgot this when testing. It doesn't hugely affect the output:

Since this seems to be a continuation of the mpv issue, thought I'd chime in here.

I've dumped the whole RPU DM metadata for the sample (and scene) in question, if it's of any use.
Every entry corresponds to the metadata of a frame.
The coefficients in there are the raw values, not the correctly interpreted/calculated ones.
So they most likely need to be calculated according to Dolby's patent.

edit: Removed old links which were the wrong metadata, see https://github.com/quietvoid/dovi_tool/tree/main/dolby_vision for parsing the RPU per frame.

mpv is BTW broken when presenting full range :(((((( it shows 0, 0, 0 as elevated in the green.

So this https://0x0.st/iREw.log is log of where you should see a change of reshaping in those screenshots https://github.com/mpv-player/mpv/issues/7326#issuecomment-591518703 (they are now gone) in iOS_P5_GlassBlowing2_3840x2160@59.94fps_15200kbps.mp4

So were you able to find that? Also, what about patches, those should be done to ffmpeg, IMHO... Your libplacebo patch to ffmpeg is very nice, BTW.

@ValZapod so in the dolby PRU it seems they signal the coefficients for the DoviIPT->DoviLMS and DoviLMS->ICtCpLMS matrices. However, I was not able to find any sample where the matrices differ from the ones hard-coded in 3c19e385.

Do you think it's important for us to make these matrices dynamic, or are we fine to just hard-code them? I suspect that no samples actually exist anywhere where these matrices would be different.

important for us to make these matrices dynamic, or are we fine to just hard-code them?

The default is anyway mandated by the patent. See: https://patentimages.storage.googleapis.com/ec/46/74/f995872532294c/US10701399.pdf see signal_color_space and above.

mpv is BTW broken when presenting full range :(((((( it shows 0, 0, 0 as elevated in the green

I think I was wrong. Since discovering -vf extractplanes genious written by Paul IMHO it is just what it is. It is correct, while VLC and Movies & TV do not support full range. Alas.

BTW, ffmpeg now parses RPU. Which is nice, @quitvoid added it all.

Also here is profile 8 DV with changing metadata (in patent above trim_slope, trim_offset, trim_power, trim_chroma_weight, trim_saturation_gain, ms_weight) on same Y'CbCr data (not RGB, LMS, IPT stuff). It works on LG C9 and Movies & TV. https://drive.google.com/file/d/1-8C0ZYnIhEqOVJQ532hIJobP5XR6A7gj/view?usp=sharing

mentioned in commit fb30b848

mentioned in commit 5bbaccef

mentioned in commit ddc6f6fd

mentioned in commit a44d6628

mentioned in merge request !207 (merged)

The default is anyway mandated by the patent. See: https://patentimages.storage.googleapis.com/ec/46/74/f995872532294c/US10701399.pdf see signal_color_space and above.

Hmm. That one is a completely different matrix than what is used in practice, though.

than what is used in practice

The default is YCbCr as used in Blu-ray. You first read signal_color_space and only then...

So did you get the reshaping part? After talking to igorzep he said the matrix does not change but it changes by some polinomial. I did not get further.

mentioned in commit 02373ca6

Implemented MMR reshaping as well (needed for profiles 8.X), not yet tested. Also implemented loading/applying custom matrices from the RPU, rather than hard-coding them.

mentioned in commit bc5f7584

mentioned in commit ff6ea7b5

mentioned in commit 2d188994

mentioned in commit ecc36b2b

mentioned in commit 291857b8

mentioned in commit 65df86f2

mentioned in commit a8f5862c

mentioned in commit 58eda8c1

mentioned in commit 29a6129c

mentioned in commit b404e522

mentioned in commit 775a9325

closed with merge request !207 (merged)

closed with commit 775a9325

Is there a simple way to do with -vf libplacebo from ffmpeg?

Not yet, no. RPU parsing support will be added to ffmpeg soon, then we can hook it up to vf_libplacebo.

It is tecnically already there. https://patchwork.ffmpeg.org/project/ffmpeg/patch/20211126170534.2041327-1-tcChlisop0@gmail.com/

That has nothing to do with RPU parsing.
It's AVDOVIDecoderConfigurationRecord, which is a record in the MP4 container.

I mean those ones around this commit. https://github.com/FFmpeg/FFmpeg/commit/54e65aa38abb37d6af92551b7e3adf6785f631ec