vout: expand stacking mode from boolean to unsigned

This enables passing a below state in addition to currently supported
always-on-top (above) and normal. (However, none of the window provider
support it just yet).