Luminance SSIM lower on yuv444 compared to yuv420 on same CRF

While comparing various encoding settings, I found an odd difference between x264's yuv444 and yuv420 pixel formats.
Specifically, if I encode the same source file with identical CRF and settings except pixel format, the luminance (Y) SSIM value compared to the source video is lower on the yuv444 encode than the yuv420 encode. (The chroma U and V SSIM components are correctly higher, as expected from no chroma subsampling)
If I wanted to maintain the same luminance SSIM in the yuv444 encode, I would have to reduce the CRF by a theoretical 0.4 from the one used in the yuv420 encode

My intuition with CRF encoding is that the only difference between yuv444 and yuv420 encodes should be chroma subsampling - the luminance component should be practically identical at the same CRF.
Is this intuition wrong, and lower luminance SSIM is expected behaviour? Or is there a bug somewhere?
(Note: the rav1e encoder does not exhibit this behaviour - the luminance SSIM is practically identical between yuv444 and yuv420 when QP is the same)

I encoded using ffmpeg with the following commands:

# <crf> values tested with 16, 18, 20, and 22
ffmpeg -i ref.webm -colorspace bt709 -an -c:v libx264 -preset veryslow -pix_fmt yuvj444p -crf <crf> yuv444-crf<crf>.mkv
ffmpeg -i ref.webm -colorspace bt709 -an -c:v libx264 -preset veryslow -pix_fmt yuvj420p -crf <crf> yuv420-crf<crf>.mkv

And then computed SSIM with:

ffmpeg -i yuv444-crf<crf>.mkv -i ref.webm -colorspace bt709 -an -filter_complex '[0:v]format=pix_fmts=yuvj444p[dist];[dist][1:v]ssim' -f null /dev/null
ffmpeg -i yuv420-crf<crf>.mkv -i ref.webm -colorspace bt709 -an -filter_complex '[0:v]format=pix_fmts=yuvj444p[dist];[dist][1:v]ssim' -f null /dev/null

Reference (source) video: ref.webm
(Cut and cropped to reduce filesize. Problem was reproducible in longer and larger videos just the same)