Writing to encrypted path resulting in Permission denied

[tiagorvmartins]

Hey Gramine team,
Just a quick question please, such that I can to try and rule out a problem I'm currently facing,

Is it possible to write to an encrypted file using pwrite?

According to the pwrite man page:

The pwrite() function shall be equivalent to write(), except that it writes into a given position without changing the file pointer. The first three arguments to pwrite() are the same as write() with the addition of a fourth argument offset for the desired position inside the file.

So I am wondering if this is dissallowed on Gramine, because if it is, then that explains it and I need to think on a alternative to patch it.

Also as additional notes, I also tested if the fd was valid and open to write to using:

fcntl(fd, F_GETFD)

But pwrite keeps returning EACCES

Any clues what might be wrong?
Can provide more details such as manifest if needed, just trying to rule out first the availability of pwrite on encrypted files.
Thank you!

[kailun-qin]

Is it possible to write to an encrypted file using pwrite?
So I am wondering if this is dissallowed on Gramine

I think this is supported.

But pwrite keeps returning EACCES

pf_write() failure would generate -EACCESS:

gramine/libos/src/fs/libos_fs_encrypted.c

Lines 580 to 584 in 6167772

	pf_status_t pfs = pf_write(enc->pf, offset, buf_size, buf);
	if (PF_FAILURE(pfs)) {
	log_warning("pf_write failed: %s", pf_strerror(pfs));
	return -EACCES;
	}

, would you pls share the verbose Gramine log and manifest? Thanks!

[tiagorvmartins]

Sorry for the delay answering and thank you for the support,

Been trying to solve it myself, redoing same steps several times and trying different approaches as well, but couldn't find solution or the culprit yet.

The rename happens inside the enclave, even though it's an application that creates several forks I believe that scenario wouldn't cause the issue correct? I even tried to use the path encrypted with the key _sgx_mrsigner instead of _sgx_mrenclave, but same issue.

Not using encrypted files is not an option as far as i know, because it relies on untrusted memory and allowed files disallow mmap according to what i have read before here: gramineproject/graphene#433 (comment)

I also understand the insecurity of using untrusted shared memory, I am quite aware of the implications that it raises.

I am adding now here the manifest and the full debug logs as attachment (long file) as @kailun-qin requested, I hope some of you two find what is going wrong here,

Logs are somewhat long, but I believe it's better this way so you have the entire context,
Thank you again for your support and time!

Note: just as side note, this is running on docker with gsc tooling (master branch)

debug_logs.txt

[kailun-qin]

It looks like /var/lib/postgresql/data/pg_wal/000000010000000000000001 was accessed concurrently by multiple processes but there is known limitation on this, pls see:

gramine/Documentation/devel/encfiles.rst

Lines 500 to 503 in e14e186

	- There is *limited* multiprocess support for encrypted files. This means that
	if the same file is accessed concurrently by two Gramine processes (and at
	least one process writes to the file), the file may become corrupted or
	inaccessible to one of the processes.

.

[tiagorvmartins]

Thank you @kailun-qin , what would be the best approach to mitigate the issue?
Does using a file lock system using flock (fcntl primitive) on top of that file could suffice across those multiple processes using the same file?

[kailun-qin]

Yes, I think file locking for serialized access would work. Pls see the details w/ caveats below:

gramine/Documentation/devel/features.md

Lines 2225 to 2258 in a9d08fd

	#### File locking
	File locking operations can be considered one of the IPC mechanisms, as discussed in ["Overview of
	Inter-Process Communication (IPC)" section](#overview-of-inter-process-communication-ipc). Thus,
	file locks are implemented via message passing in Gramine, and all lock-requests are handled in the
	main (leader) process.
	Gramine currently implements two types of file locks:
	- POSIX (fcntl) locks aka Advisory record locks. In particular, the following operations are
	implemented: `fcntl(F_SETLK)`, `fcntl(F_SETLKW)` and `fcntl(F_GETLK)`.
	- BSD (flock) locks. The following system call is implemented: `flock()`. Its support is currently
	experimental and not suitable for production.
	Both types of file locks share the same internal implementation in Gramine. The current
	implementation has the following caveats:
	- Lock requests from other processes will always have the overhead of IPC round-trip, even if the
	lock is uncontested.
	- The main process has to be able to look up the same file, so locking will not work for files in
	local-process-only filesystems (e.g. tmpfs).
	- There is no deadlock detection (`EDEADLK`). This is only applicable to POSIX locks; BSD locks do
	not have deadlock detection in the first place.
	- The lock requests cannot be interrupted (`EINTR`).
	- The locks work only on regular files (no pipes, sockets etc.).
	Similarly to Linux, BSD (flock) locks ignore deprecated `LOCK_{MAND,READ,WRITE,RW}` operations.
	BSD (flock) locks are currently experimental and are *disabled* by default. To enable them, use the
	`sys.experimental__enable_flock` {ref}`manifest option
	<experimental-flock-bsd-style-locks-support>`. There is at least one problem with BSD locks
	currently: they are supposed to be released when the last reference (file descriptor, or FD) to the
	underlying opened file is closed, including when a process with the opened file terminates.
	Unfortunately, Gramine lacks system-wide tracking of opened files' FDs. This may lead to premature
	releases of flock locks in some situations.

.