dav1d: treat RGB formats separately

This avoids incorrectly matching as GBR. Fixes #28763 (closed) which was missing checks for the matrix and primaries.

This also replaces the loop with a constant-time lookup.

(cherry picked from commit ece317a2)