YCbCr 160, 160, 160 must decode as 225, 144, 235 using BT.709 matrix (and is decoded as such by OpenGL for windows in VLC and by mpv) but Directx 11 VideoProcessorBlt is broken if you are using Intel (decodes as 225, 143, 235), that YOU do, see #16193. But with Nvidia's VideoProcessorBlt it is working correctly, i.e. it decodes as 225, 144, 235.
Do not forget that ffmpeg requires -vf scale parameters as the default is BT.601 matrix and tag it as BT.709 with -colorspace (both are needed, second will not affect the first):
We want full support including primaries color managment, WIDE-gamut color managment, icc managment and BT.1886 EOTF. HDR/SDR imagery in AVIF also needs it. It is a giant show that I started and some other people.
I don't understand you're question, we don't use any VideoProcessBlt here. We only use it when manually enabling video filters. And when we use it we don't ask for any color conversion, i.e. the input is the same as the output.
So you are saying the only possible problem is in HLSL code above? my fix cannot change anything there (for 8 bit), since 8 bit is just 0.004 of difference...
input is the same as the output.
Well, it is converted somewhere to R'G'B', no change of transfer function is needed, BTW.
Oh, and BTW matrix coeff. in HLSL are wrong, i.e. they are BT.709-1, nobody used them, everybody switched to BT.709-2. I can pull a fix. Not that it should affect anything for 8 bits. See https://en.wikipedia.org/wiki/Rec._709#Luma_coefficients
My code change is only 0.0002 of difference in float, while 8 bit is only 0.004 which is 10 times less. If it would have affected something for 8 bit, it would have been bad.
Okay, looks like you first decode to limited range R'G'B', then round and then convert to full range R'G'B'. That will give that bad value. That is a bad idea, of course. Y'Cb'Cr' is much bigger colorspace than R'G'B' and has less gradations in R'G'B' part of it, so by decoding to limited range R'G'B' you lose some possible precision.
And anyway. You are supposed to opperate on float anyway and only quantise in the end (whether it is integer optimisation or not).
Interesting, then why they fixed HDR black levels and range in next driver all 3 times? (I mean besides support for LG TV, which are really monitors with 3DLUT.) Do you have a link on https://forums.developer.nvidia.com/search?q=StretchRect?