access/vnc: vnc pixfmt fixes
This merge request was primary motivated by fixing rendering on little-endian clients. Outside of keeping bits per pixel consistent, we never made any attempt to match the server's native pixel format, always requesting our own hardcoded format for any color depth, so we might as well also always ask the server for LE byte order (regardless of client endianness) and let our code simply assume that. This change has been tested with LE and BE clients. Previous to this commit colors were wrong in the case of (at least) LE <-> LE clients and servers.
In investigating the above issue, I also discovered and fixed two other issues. The first relates to how libvncclient does not simultaneously support 8 bpp with tight encoding, so if we advertise support for this encoding we should ask the server to upgrade the color depth from 8 to 16 so that we do not get something that we will ultimately not be able to decode.
The second issue relates to the code conflating color depth with bpp, as these are two separate values communicated between server and client and are not necessarily the same. Servers based on libvncserver at least seem to just ignore the depth
field and only look directly at redShift
, redMax
, etc., which is probably why this has never culminated in a noticeable bug, but we should still transmit and interpret these values correctly.