vulkan/malloc: refactor and fix host-visible VRAM limit
The implementation of this limit had some serious problems:
-
It was a violation of layers, since it was conceptually part of the malloc layer but checked by vulkan/gpu_buf.c
-
It failed to account for the size increase caused by slab paging, the minimum number of which is 4 pages per slab, leading to an unintended 4x size increase of the actual visible VRAM usage - so the limit was not 1/64th of VRAM as believed, but 1/16th at best, and possibly 100% at worst!
-
It failed to exclude possible other sources of large host-visible VRAM allocations, such as from textures etc.
-
It was needlessly punishing machines where there is no limit on host- visible VRAM, such as when ReBAR is enabled, by disabling the use of host-visible VRAM for large allocations even when there is no penalty associated with them.
Fix this by moving the check to the malloc layer itself, and checking each allocation at the time of that specific allocation itself. Possibly, we could also track how much of VRAM we are using overall, and put a hard limit on the total usage. But for now, this check is already much better than the status quo.