The current logic could end up with a buffer that needs to be both
device-local (for vertex information) and host-visible (for updates).
This is obviously nonsensical, so work around this case instead.

The ideal solution is to use a separate staging buffer, but that
requires more logic changes than this workaround, and the only
difference is that this might be possibly slower.