Porting Vault in Gramine using Raft storage type returns memory allocation error

[dzobbe]

Hi All,

I am trying to port Hashicorp Vault in Gramine. When using the Vault storage configuration file everything works fine. However, I need to replication feature of Vault and to do so I need to use a storage configuration raft.
This storage configuration needs flock and in fact when I enabled it inside the manifest it went through the initial error related to flock. But a new error came up: cannot allocate memory

Manifest:

libos.entrypoint = "/usr/bin/vault"
loader.argv = [ "/usr/bin/vault", "server", "-config", "config.hcl" ]

loader.entrypoint = "file:{{ gramine.libos }}"

loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}"
loader.env.HOME = "/root"

loader.log_level = "all"
#loader.log_file = "log.txt"

fs.root.uri = "file:/"
fs.start_dir = "/app"

fs.mounts = [
    { path = "/data", uri = "file:/data", type = "encrypted", key_name = "default" },
]
fs.insecure__keys.default = "ffeeddccbbaa99887766554433221100"

sgx.allowed_files = [
  "file:/",
]

sys.experimental__enable_flock = true

sgx.max_threads = 16
sgx.enclave_size = "4G"
sgx.debug = true
sgx.nonpie_binary = true

Vault config:

ui            = true
cluster_addr  = "https://127.0.0.1:8201"
api_addr      = "https://127.0.0.1:8200"
disable_mlock = true

log_level = "Debug"

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}

storage "raft" {
  path    = "/data"
  node_id = "node0"
}

vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T1:vault] trace: ---- futex(0x140d4548, FUTEX_PRIVATE|FUTEX_WAKE, 1, 0, 0, 0) ...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T1:vault] trace: ---- return from futex(...) = 0x1
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from futex(...) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- clock_gettime(1, 0x140e5db0) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T1:vault] trace: ---- futex(0x10f9b788, FUTEX_PRIVATE|FUTEX_WAIT, 0, 0, 0, 0) ...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- uname(0x176fe93a) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- newfstatat(AT_FDCWD, "/data/vault.db", 0x16e8f6f8, 0) = -2
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- newfstatat(AT_FDCWD, "/data/vault.db", 0x16e8f7c8, 0) = -2
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- clock_gettime(0, 0x140e5fb0) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- clock_gettime(1, 0x140e5fa0) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- openat(AT_FDCWD, "/data/vault.db", O_RDWR|O_CREAT|0x80000, 0600) = 0x3
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- fcntl(3, F_GETFL, 0) = 0x80042
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- fcntl(3, F_SETFL, 0x80842) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- epoll_ctl(4, ADD, 3, {.events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, .data=0x194a2c200007}) = -1
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- fcntl(3, F_GETFL, 0) = 0x80842
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- fcntl(3, F_SETFL, 0x80042) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- clock_gettime(0, 0x140e5fb0) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- clock_gettime(1, 0x140e5fa0) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- flock(3, LOCK_EX|LOCK_NB) ...
vault_1  | (libos_fs_lock.c:112:file_lock_dump_write_all) [P1:T3:vault] file_lock:  flock (BSD): handle id=4294967321: w
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from flock(...) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- fstat(3, 0x16e8f898) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- pwrite64(3, 0x16f4a000, 0x4000, 0x0) = 0x4000
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- fdatasync(3) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- fstat(3, 0x16e8f968) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- mmap(0, 0x1900000000, PROT_READ, MAP_SHARED, 3, 0x0) ...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from mmap(...) = -12
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- flock(3, LOCK_UN) ...
vault_1  | (libos_fs_lock.c:112:file_lock_dump_write_all) [P1:T3:vault] file_lock: no locks
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from flock(...) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- close(3) = 0x0
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- write(1, 0x16b9ba00, 0x6f) ...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from write(...) = 0x6f
vault_1  | Error initializing storage of type raft: failed to create fsm: failed to open bolt file: cannot allocate memory(libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- write(1, 0x16cfb100, 0x1) ...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from write(...) = 0x1
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- write(2, 0x16c8ab40, 0x5d) ...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from write(...) = 0x5d
vault_1  | (libos_exit.c:212:libos_syscall_exit_group) [P1:T3:vault] debug: ---- exit_group (returning 1)
vault_1  | 
vault_1  | 2023-10-06T13:27:22.946Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T1:vault] trace: ---- return from futex(...) = -512
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T4:vault] trace: ---- return from epoll_pwait(...) = -514
vault_1  | (libos_init.c:561:create_pipe) [P1:T4:vault] debug: Creating pipe: pipe.srv:5b2c43c6a12db9577b7ba7e1a5504aa9fa0c93ba27409f6c7094cf5d35050024
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T8:vault] trace: ---- return from futex(...) = -512
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T2:vault] trace: ---- return from nanosleep(...) = -4
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T5:vault] trace: ---- return from futex(...) = -512
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T6:vault] trace: ---- return from futex(...) = -512

[kailun-qin]

@dzobbe Thanks for the report!

Pls note that from the Gramine log,

vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- openat(AT_FDCWD, "/data/vault.db", O_RDWR|O_CREAT|0x80000, 0600) = 0x3
...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- mmap(0, 0x1900000000, PROT_READ, MAP_SHARED, 3, 0x0) ...
vault_1  | (libos_parser.c:1628:buf_write_all) [P1:T3:vault] trace: ---- return from mmap(...) = -12
...
vault_1  | Error initializing storage of type raft: failed to create fsm: failed to open bolt file: cannot allocate memory

vault was trying to create a huge (length 0x1900000000 -- 100GB) file mapping from encrypted file /data/vault.db and failed w/ ENOMEM (-12) as the enclave size was only 4GB (sgx.enclave_size = "4G", EDMM disabled, from your manifest).

I think one solution would be to set VAULT_RAFT_INITIAL_MMAP_SIZE env variable to a reasonable value and also adjust your enclave size accordingly, pls see https://github.com/hashicorp/vault/blob/61f8aff24b4d2e339abc1bc32115bf1fe39bcc26/physical/raft/raft.go#L1900-L1915 for details.

[dzobbe]

Yeah @dimakuv I also don't think it is a matter of bug but more like the involved libraries are doing something that Gramine is not fit with. The "guilty" is bbolt . Their memory management is what is actually causing a lot of issues:

_"Bolt uses a memory-mapped file so the underlying operating system handles the caching of the data. Typically, the OS will cache as much of the file as it can in memory and will release memory as needed to other processes. This means that Bolt can show very high memory usage when working with large databases. However, this is expected and the OS will release memory as needed. Bolt can handle databases much larger than the available physical RAM, provided its memory-map fits in the process virtual address space. It may be problematic on 32-bits systems.

The data structures in the Bolt database are memory mapped so the data file will be endian specific. This means that you cannot copy a Bolt file from a little endian machine to a big endian machine and have it work. For most users this is not a concern since most modern CPUs are little endian.

Because of the way pages are laid out on disk, Bolt cannot truncate data files and return free pages back to the disk. Instead, Bolt maintains a free list of unused pages within its data file. These free pages can be reused by later transactions. This works well for many use cases as databases generally tend to grow. However, it's important to note that deleting large chunks of data will not allow you to reclaim that space on disk."_

I tried your suggestions but still the same. I will try to understand if that can be tuned somehow or otherwise I just need to give up :(

[monavij]

You may try this app with EDMM enabled, so you don't have to build the whole enclave at startup. With EDMM the enclave will be able to grow dynamically.

[dzobbe]

Thank you @monavij . I tried that as well but no luck.
I get memory allocation error if I don't set VAULT_RAFT_INITIAL_MMAP_SIZE to zero or to some low values (e.g., 2GB).
Otherwise, I get again the runtime error: invalid memory address or nil pointer dereference.

[dzobbe]

After further investigations I discovered that my issue with bbolt (underlying the crash of Vault) was something similar to this one. I tried using this PR #1523 and it worked fine. I hope it will be merged soon :)

Thanks for the support.