execute GSC with encrypted data - error failed to decrypt metadata

[tiagorvmartins]

I am currently setting up two docker containers, following the examples of ra-tls-secret-prov:

• one that serves as a local server secret provisioning through dcap attestation
• another that is the client and uses a graminized sgx image

I am mounting a volume which contains several encrypted files, that were previously encrypted with:
gramine-sgx-pf-crypt encrypt -w wrap_key -i $file -o "enc_files/$file"

Logs of the server secret provisioning:

--- Reading the master key for encrypted files from '/server/wrap_key' ---
--- Starting the Secret Provisioning server on port 4433 ---Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
WARNING: The collateral is out of date.

Logs of the gsc sgx image:
logs here

Anyway the important pieces of the logs are here below (I believe)

(libos_parser.c:1653:buf_write_all) [P1:T8:geth] trace: ---- getrandom(0x1384b3438, 0x8, 0) = 0x8
(libos_parser.c:1653:buf_write_all) [P1:T8:geth] trace: ---- newfstatat(AT_FDCWD, "/root/.ethereum/geth", 0x138422ac8, 0) = 0x0
(libos_fs_encrypted.c:141:cb_debug) [P1:T8:geth] debug: ipf_open: handle: 0, path: '/root/.ethereum/geth/LOCK', real size: 4096, mode: 0x3
(libos_fs_encrypted.c:124:cb_aes_gcm_decrypt) [P1:T8:geth] warning: lib_AESGCMDecrypt failed: -6
(libos_fs_encrypted.c:141:cb_debug) [P1:T8:geth] debug: ipf_init_existing_file: failed to decrypt metadata: -11
(libos_fs_encrypted.c:141:cb_debug) [P1:T8:geth] debug: ipf_open: failed: -11
(libos_fs_encrypted.c:203:encrypted_file_internal_open) [P1:T8:geth] warning: pf_open failed: Callback failed
(libos_parser.c:1653:buf_write_all) [P1:T8:geth] trace: ---- openat(AT_FDCWD, "/root/.ethereum/geth/LOCK", O_RDONLY|O_CREAT|0x80000, 0600) = -13
(libos_parser.c:1653:buf_write_all) [P1:T8:geth] trace: ---- fstat(1, 0x138422b98) = 0x0
(libos_parser.c:1653:buf_write_all) [P1:T8:geth] trace: ---- fstat(2, 0x138422c68) = 0x0
Fatal: Failed to create the protocol stack: open /root/.ethereum/geth/LOCK: permission denied

Already tried deleting the LOCK file because it can be deleted, its a leftover of the previous execution of the app outside sgx,
But then it also fails while decrypting other files like LOG and CURRENT.
Also would like to mention that without encrypting the volume files, it actually works even if those files exist in the volume being attached.

Manifest for the GSC image:

sys.enable_sigterm_injection = true

sgx.remote_attestation = "dcap"

sys.stack.size = "16M"
sys.enable_extra_runtime_domain_names_conf = true
sys.experimental__enable_flock = true
sgx.nonpie_binary = true
sgx.enclave_size = "8G"
sgx.edmm_enable = false
sgx.max_threads = 32

fs.mounts = [
  { path = "/root/.ethereum", uri = "file:/root/.ethereum",  type = "encrypted" }
]

sgx.trusted_files = [
  "file:/root/.ethereum/",
  "file:/usr/local/bin/geth",
  "file:/etc/host.conf",
  "file:/etc/hosts",
  "file:/etc/nsswitch.conf",
  "file:/etc/resolv.conf",
  "file:/serv_prov/ssl/ca.crt"
]

sgx.allowed_files = [
  "file:/root/.ethereum/",
  "file:/usr/local/bin/geth",
  "file:/serv_prov/ssl/ca.crt"
]

loader.env.TZ = ":/etc/localtime"
loader.env.SGX_AESM_ADDR="1"
loader.env.LD_PRELOAD = "libsecret_prov_attest.so"
loader.env.SECRET_PROVISION_CONSTRUCTOR = "1"
loader.env.SECRET_PROVISION_SET_KEY = "default"
loader.env.SECRET_PROVISION_CA_CHAIN_PATH = "/serv_prov/ssl/ca.crt"
loader.env.SECRET_PROVISION_SERVERS = "secretprovisioning:4433"

Already tried clearing all the docker image caches with prune and rebuilding everything from scratch,
Is there anything in the logs that hint to something that I might be doing wrong?

Thank you!

[dimakuv]

Several notes

Encrypted vs trusted vs allowed files are mutually exclusive

You must not specify the same file/directory in more than one of sgx.trusted_files, sgx.allowed_files and fs.mounts = { type="encrypted" } manifest keys.

In your manifest example, you have /root/.ethereum directory specified in all three. This is undefined behavior (unfortunately, Gramine currently doesn't have the logic to double-check the sanity of the manifest).

Similarly, you have e.g. /usr/local/bin/geth in both trusted files and allowed files -- again, undefined behavior. Please keep only in one of them.

Does your client really get the encryption key?

I see that you specify loader.env.SECRET_PROVISION_SERVERS = "secretprovisioning:4433" in the client app manifest. Is this secretprovisioning a real domain name for your other Docker container? Are you sure that the client successfully connects to the secret provisioning service in the other Docker container? (You can check the output of the service to make sure that the client indeed connects and the service indeed sends the encryption key to the client.)

Well, looking at your client log, it feels like yes, the client gets the correct encryption key. But please double-check.

Encrypted files must not be intermingled with regular plaintext files

Already tried deleting the LOCK file because it can be deleted, its a leftover of the previous execution of the app outside sgx

That's correct. You cannot have a plaintext file on your host Linux, and access it via Encrypted Files in Gramine. Gramine will detect that the file exists but is NOT encrypted, and will fail. TLDR: Encrypted files must either not exist (and the application creates them) or they must be truly encrypted.

Encrypted files must have a correct filename

But then it also fails while decrypting other files like LOG and CURRENT.

Encrypted Files crypto protocol in Gramine has a special feature: the file must have the same name as it was encrypted with gramine-sgx-pf-crypt encrypt. Our gramine-sgx-pf-crypt tool even has a warning about this naming peculiarity. So you should encrypt with exactly the same name as will be used by the application.

This may be the root cause of your "failed to decrypt" error.

[tiagorvmartins]

Hello @dimakuv again thank you for all the help and detailed explanations.

The problem was indeed in the encrypted file name, I was encrypting to enc_files/path/path2/path3/filename.
And then even though I was mounting on the expected path for the program to run, the encryption filename was different, because in the mounting there wasn't "enc_files" which was in the encrypted filename metadata.

After adjusting with started working flawlessly. Also adjusted the:
Encrypted vs trusted vs allowed files are mutually exclusive
Thank you for that detail!

I do have two remaining questions if you could answer, thank you.
These files are being encrypted outside and being sent to the gsc volume, which are then decrypted transparetly by gramine - all good here.

1. let's say I mount the same volume with encrypted type but empty volume, does new files that are created inside that path are encrypted as well?

2. how is the key being chosen for decrypting the files on runtime? the secret provisioning server is launched with a single key file as parameter for the server - like the example here, I don't see anywhere in the manifest sgx of the client any relation to the key file name being used. I believe my ultimate question to this would be - would be possible to decrypt different volumes using different key files?

At documentation:
The key_name mount parameter specifies the name of the encryption key. If omitted, it will default to "default". This feature can be used to mount different files or directories with different encryption keys.

• this "default" key where is being specified on the secret provisioned server?

I am asking this because I might need to adapt the server.c to my own needs.
Thank you!

[dimakuv]

let's say I mount the same volume with encrypted type but empty volume, does new files that are created inside that path are encrypted as well?

Yes. As long as the directory at which you mount your volume is specified as fs.mounts = [ {type = "encrypted", ...} ], then Gramine transparently encrypts/decrypts all files under this directory, including newly created ones.

how is the key being chosen for decrypting the files on runtime? the secret provisioning server is launched with a single key file as parameter for the server - like the example here, I don't see anywhere in the manifest sgx of the client any relation to the key file name being used.

Yes, Gramine allows to use different encryption keys for different mounted volumes. The manifest syntax example can be seen here (search for key_name):
https://github.com/gramineproject/gramine/blob/master/libos/test/regression/manifest.template

But you are right -- our ra-tls-secret-prov example is very limited, and it only contains a single encryption key (stored in a single key file), and it sends this single key to the Gramine-SGX client. The key name for this sent key is specified in SECRET_PROVISION_SET_KEY in the manifest file, and it is "default" in your case, which corresponds to the default key_name for FS mounts in the manifest file. See documentation (search for "default"):

• https://gramine.readthedocs.io/en/stable/manifest-syntax.html#encrypted-files
• https://gramine.readthedocs.io/en/stable/attestation.html#high-level-secret-provisioning-interface

I believe my ultimate question to this would be - would be possible to decrypt different volumes using different key files?

Yes, but you'll need to modify the ra-tls-secret-prov example a bit, to allow more than one key file (both on the server and on the client).

this "default" key where is being specified on the secret provisioned server?

No, our Secret Provisioning server example (server.c) simply sends a single key it possesses to the Gramine-SGX client. The client receives this single key and names it as whatever is specified in the SECRET_PROVISION_SET_KEY environment variable.

I am asking this because I might need to adapt the server.c to my own needs.

That's right. But if you want to send several keys to the Gramine-SGX client, you'll also need to modify the client to be able to receive more than one key. I recommend you to look at a more complicated example: https://github.com/gramineproject/gramine/tree/master/CI-Examples/ra-tls-secret-prov/secret_prov. This example allows to send/receive arbitrary data, so you can create your own protocol to send a bunch of keys on the server and receive a bunch of keys on the client. And when the client receives each key, the client installs this key inside Gramine via /dev/attestation/keys/<key_name>.

[tiagorvmartins]

I believe I understood now, I would need to modify my client to receive each key on the client from the server, and then store them on the path /dev/attestation/keys/<key_name> before any attempting to a file read of the encrypted files which are using encrypted <key_name>.

To do this: since I have as an entrypoint a graminized application binary I would need to either change that binary code itself to get those secrets <key_name> and save them on each path on startup, or using a bash script entrypoint which would run these two applications in order:

1. to get the secrets and save them on /dev/attestation/<key_name>
2. launch the actual app binary which uses the encrypted volume mounts with <key_name>

Can you confirm this would be the way going forward?
Accepting answer anyway.
Thank you!

[dimakuv]

I believe I understood now, I would need to modify my client to receive each key on the client from the server, and then store them on the path /dev/attestation/keys/<key_name> before any attempting to a file read of the encrypted files which are using encrypted <key_name>.

This is exactly correct.

To do this: since I have as an entrypoint a graminized application binary I would need to either change that binary code itself to get those secrets <key_name> and save them on each path on startup, or using a bash script entrypoint which would run these two applications in order ...

Yes. You can change the application code itself, to receive and set up the encryption keys at startup - but I wouldn't recommend this path.

You can also start Bash as your "preliminary setup" script, and then from Bash execute your actual application.

But I would recommend to write a tiny C program that would act as the "preliminary setup" binary:

1. This C program is specified as the Gramine manifest entrypoint.
2. This C program connects to the Secret Prov server, receives the keys and installs them in /dev/attestation/keys/.
3. Finally, this C program calls execl("your-actual-app", your-actual-cmdline-args, ...).

This is a so-called pre-main approach. It has two benefits:

• No need to modify the original app.
• No need to bring in a slow-and-heavy Bash session into Gramine.

[tiagorvmartins]

I understand, makes total sense, thank you again for all the help!