RGB mask endianness using is bogus/inconsistent

The i_rmask, i_bmask, i_bmask of the video format don't specify whether they are using native endianness or values meant to match the memory order of the color components.

The default mask for VLC_CODEC_RGB32 is 0x00ff0000, 0x0000ff00, 0x000000ff, according to video_format_FixRGB.

The libavutil code maps it to AV_PIX_FMT_0BGR when compiled on Little-Endian and AV_PIX_FMT_0RGB on Big-Endian:

VLC_RGB( VLC_CODEC_RGB32, AV_PIX_FMT_0RGB, AV_PIX_FMT_0BGR, 0x00ff0000, 0x0000ff00, 0x000000ff)

This suggests that the mask are not supposed to match the memory order.

We have code like GetPackedRgbIndexes that is supposed to give the index in memory order given a chroma+mask. It also gives XBGR/0BGR on Little-Endian for the default mask.

So far, so good.

But there's also the RGB to I420 converter that assumes different things:

    /* If we don't have support for the bitmasks, bail out */
    if( p_filter->fmt_out.video.i_rmask == 0x00ff0000
        && p_filter->fmt_out.video.i_gmask == 0x0000ff00
        && p_filter->fmt_out.video.i_bmask == 0x000000ff )
    {
        /* A8R8G8B8 pixel format */
        msg_Dbg(p_filter, "RGB pixel format is A8R8G8B8");
        p_filter->ops = &I420_A8R8G8B8_ops;
    }

The mask is also used to read/write RGB from AVI and to set the mask in wingdi for RGB. The usage of wingdi can help us find what mask should be written in AVI (BITMAPINFO).

I modified the mock code to set 4 primary colors in 4 vertical division:

#if 1
    uint32_t *pixels = (uint32_t*)pic->p[0].p_pixels;
    unsigned lines_per_color = pic->p[0].i_visible_lines / 4;
    for (unsigned y=0*lines_per_color; y < 1*lines_per_color; y++)
        for (int x=0; x < pic->p[0].i_visible_pitch / 4; x++)
        {
            pixels[x + y*pic->p[0].i_pitch/4] = hton32(0xFF000000);
        }
    for (unsigned y=1*lines_per_color; y < 2*lines_per_color; y++)
        for (int x=0; x < pic->p[0].i_visible_pitch / 4; x++)
        {
            pixels[x + y*pic->p[0].i_pitch/4] = hton32(0x00FF0000);
        }
    for (unsigned y=2*lines_per_color; y < 3*lines_per_color; y++)
        for (int x=0; x < pic->p[0].i_visible_pitch / 4; x++)
        {
            pixels[x + y*pic->p[0].i_pitch/4] = hton32(0x0000FF00);
        }
    for (unsigned y=3*lines_per_color; y < 4*lines_per_color; y++)
        for (int x=0; x < pic->p[0].i_visible_pitch / 4; x++)
        {
            pixels[x + y*pic->p[0].i_pitch/4] = hton32(0x000000FF);
        }
#else
    memset(pic->p[0].p_pixels, pixel, block_len);
#endif

When using this with mock://video_track_count=1;video_chroma=RV32 the default RGB32 mask ends ups showing Blue, Green, Red, Black from top to bottom. This corresponds to BGRX, and not XBGR as expected.

When using mock://video_track_count=1;video_chroma=BGRX and forcing VLC_CODEC_BGRX in wingdi, the proper mask to use is:

    *((DWORD*)&sys->bi_rgb.red)   = hton32(0x0000ff00);
    *((DWORD*)&sys->bi_rgb.green) = hton32(0x00ff0000);
    *((DWORD*)&sys->bi_rgb.blue)  = hton32(0xff000000);

This is the opposite of the masks we are using right now. In bitmapheaderinfo we use

    SetDWBE( &p_bmiColors[0], i_rmask );
    SetDWBE( &p_bmiColors[4], i_gmask );
    SetDWBE( &p_bmiColors[8], i_bmask );

In this case the mask values should not use hton32(), 5c42f091 is wrong.