Huge pages are tricky and maybe even buggy

[jounathaen]

I think, the current use of huge pages might have some issues.

First some background:
On Linux, there are apparently two ways of allocating huge pages:

• Static Huge Pages: These need to be allocated at boot or via the cli before running uhyve, and there is only a limited number of them. You can check this number with cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
  • on my system, this number is actually pretty small. Far too small for proper use with more than a few MiB
• Transparent Huge Pages: The kernel allocates pages on huge pages if possible. This is activated for a memory region with madvise.
  • Note that this does not modify previously allocated pages.
  • This relies on standard huge pages (check with cat /proc/meminfo | grep 'AnonHugePages:')

I've stumbled upon two issues:

• mmap has no way to specify the alignment of the memory. So our MmapMemory struct cannot guarantee that the memory is located at a huge page boundary. I'm not sure if this really is an issue, but I think that it might be suboptimal to have huge pages in the vm being mapped to normal pages.
• if the MAP_HUGETLB flag is passed to mmap, but there are no huge pages available, things just start to break without a warning. (The mmap does not fail, but accessing the memory results in SIGBUS).

Possible solutions:

• We could add a check if the huge page was actually loaded to memory using mincore: https://stackoverflow.com/questions/37954675/getting-sigbus-after-mmap-for-huge-pages-even-though-hugepages-free-is-positive
  • This feels like an odd hack, and this effectively changes the lazy-loading of the memory to preloading, probably resulting in higher memory usage and overhead.
• We could at least allocate the memory aligned to huge pages using aligned_alloc
  • I have not found a Rust abstraction for this call yet.