Skip to content

picture: fix visible pitch/lines initialization

Romain Vimont requested to merge rom1v/vlc:fix_plane_visible_area into master

A video_format_t defines (in pixels):

  • a picture size (i_width, i_height);
  • a visible rectangle area (i_visible_width, i_visible_height, i_x_offset, i_y_offset).
                   i_width
              <---------------->
            ^ +----------------+ ^
            | |                | | i_y_offset
            | |                | v
            | |  ...........   | ^
   i_height | |  : visible :   | |
            | |  :  area   :   | | i_visible_height
            | |  ...........   | v
            | |                |
            v +----------------+
              <-><--------->
               |  i_visible_width
               |
           i_x_offset

A picture_t contains an array of plane_t, which defines:

  • the physical storage size (i_pitch, i_lines);
  • a content area (i_visible_pitch, i_visible_lines).

A picture_t may be created and allocated from a video_format_t via picture_NewFromFormat(). This function calls picture_InitPrivate(), which itself calls picture_Setup(); here, the pitch and lines fields are computed from aligned versions of the format i_width and i_height (to at least a multiple of 64 or 32 respectively). Therefore, the resulting picture storage may be larger than the size defined in the video_format_t.

This is why a content area is necessary in plane_t to store the actual content size, and this explains why an offset is not defined (an alignment may only add margins to the right and to the bottom).

                   i_width
              <---------------->
            ^ +----------------+*** ^                 ^
            | |                |*** |                 |
            | |                |*** | i_visible_lines |
            | |  ...........   |*** |                 |
 i_height   | |  : visible :   |*** |                 |
            | |  :  area   :   |*** |                 | i_lines
            | |  ...........   |*** |                 |
            | |                |*** |                 |
            v +----------------+*** v                 |
              *********************                   |
              *********************                   v
              <---------------->
                i_visible_pitch         *: margins for alignment
                  (in bytes)

The fields defining the content area in plane_t are misleadingly called "visible", but they do not represent the "visible" area: they define the actual format size (i_width and i_height) before alignment.

Concretely, for p a plane_t and fmt the video_format_t from which it is created:

  • p->i_pitch = align(fmt->i_width) * pixel_size
  • p->i_lines = align(fmt->i_height)
  • p->i_visible_pitch = fmt->i_width * pixel_size
  • p->i_visible_lines = fmt->i_height

Due to this confusion, p->i_visible_pitch and p->i_visible_lines were mistakenly initialized from fmt->i_visible_width and fmt->i_visible_height.

Fixes #27264 (closed)

Edited by Romain Vimont

Merge request reports