Failed to connect secret provisioning with Intel SGX and Gramine-PyTorch

[mrinal054]

Hello,

I was doing PyTorch privacy-preserving machine learning applications with Intel SGX and Gramine, as described in Gramine’s GitHub.

I was able to execute the deep learning model inside the enclave without remote attestation.

However, I am facing problems establishing the remote attestation. Here is the error I am receiving when I am trying to connect to the DCAP server -

~/Gramine_project/examples/pytorch-confidential$ gramine-sgx ./pytorch pytorchexample.py
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

secret_provision_start: Secret Provisioning could not connect to any of the servers specified in SECRET_PROVISION_SERVERS; last mbedTLS error was -68
secret_provision_common_close: Secret Provisioning failed during connection close with mbedTLS error -28928
WARNING: Closing the secret-prov context failed with error -1.
secret_provision_constructor: Secret provisioning failed, terminating the whole process

Here is my server side -

Here is my user side -

I have changed the manifest file as instructed by Gramine. Here is the changes that I made to established the connect and DCAP -

sgx.trusted_files = [
  "file:{{ entrypoint }}",
  "file:{{ gramine.libos }}",
  "file:{{ gramine.runtimedir() }}/",
  "file:/usr/lib/",
  "file:{{ arch_libdir }}/",
  "file:/usr/{{ arch_libdir }}/",
{% for path in python.get_sys_path(entrypoint) %}
  "file:{{ path }}{{ '/' if path.is_dir() else '' }}",
{% endfor %}

  "file:pytorchexample.py",
  "file:ssl/ca.crt",
]

sys.enable_extra_runtime_domain_names_conf = true
sgx.remote_attestation = "dcap"
loader.env.LD_PRELOAD = "libsecret_prov_attest.so"
loader.env.SECRET_PROVISION_CONSTRUCTOR = "1"
loader.env.SECRET_PROVISION_SET_KEY = "default"
loader.env.SECRET_PROVISION_CA_CHAIN_PATH = "ssl/ca.crt"
loader.env.SECRET_PROVISION_SERVERS = "localhost:4433"

I am using Gramine 1.5. Here is the version details -

Package: gramine
Version: 1.5
Priority: optional
Section: misc
Maintainer: Wojtek Porczyk <woju@invisiblethingslab.com>
Installed-Size: 21.9 MB
Depends: libcurl4 (>= 7.58), libprotobuf-c1, python3, python3-click, python3-cryptography, python3-jinja2, python3-pyelftools, python3-tomli (>= 1.1.0), python3-tomli-w (>= 0.4.0)
Recommends: gramine-ratls-dcap, gramine-ratls-epid
Conflicts: gramine-oot
Breaks: gramine-dcap (<< 1.4~)
Replaces: gramine-dcap (<< 1.4~)
Homepage: https://gramine.readthedocs.io/
Download-Size: 3,351 kB
APT-Manual-Installed: yes
APT-Sources: https://packages.gramineproject.io jammy/main amd64 Packages
Description: A lightweight usermode guest OS designed to run a single Linux application

Please note that I have also tried with setting the RA_TLS_ALLOW_OUTDATED_TCB_INSECURE to 1.

As the complete Gramine log is quite long, I am attaching a text file that contains the entire log for loader.log_level = "all".

log_detail.txt

Could you please suggest how to resolve the issue?

[dimakuv]

Hm, a bit strange.

Can you add the following item to the sgx.trusted_files list:

sgx.trusted_files = [
   ...
   "file:/etc/hosts",
]

and then rebuild the PyTorch example: make clean; make SGX=1 and try again?

[kailun-qin]

Thanks!

secret_provision_start: Secret Provisioning failed during mbedtls_ssl_handshake with error -9984

-9984 (-0x2700) indicated that the certificate verification failed, e.g. CRL, CA or signature check failed (pls see: https://github.com/Mbed-TLS/mbedtls/blob/1fdc884ed831927600838a3656b6d74a0417d52b/include/mbedtls/x509.h#L83).

However, there are several reasons for certificate verification to fail (e.g., incorrect trusted root CA certificate, the server hostname is different from the certificate CN in SN\SAN...). Since you have both the client and server running on the same localhost and you are able to run ra-tls-secret-prov example correctly and you directly copied the related files, I don't see anything suspicious obviously and can't tell w/o further debug info. Any other context I may have missed?

[dimakuv]

@keke-chen This is suspicious in your output:

client_connection: ra_tls_verify_callback_results:
    attestation_scheme=0, err_loc=0, 
client_connection:     unknown attestation scheme!

Somehow the secret provisioning server (./server_dcap) doesn't understand the your Gramine-SGX PyTorch client sent a DCAP attestation quote. Are you sure you have a "classic" SGX attestation infrastructure? Could your SGX infrastructure be based not on Intel DCAP, but on something else (like Microsoft Azure Attestation)?

[kailun-qin]

@dimakuv I think this can happen when mbedtls handshake failed early and not even reached the RA-TLS verify callback (i.e., irrelevant to the attestation scheme)?

[keke-chen]

Here is the output and log of successfully running gramine/CI-Examples/ra-tls-secret-prov/secret_prov_pf
It's on the same server -- I copied server_dcap and the ssl/ directory to pytorchconfidential as instructed

=====
gramine-sgx ./client 2>err1.txt
Allowing quote status CONFIG_AND_SW_HARDENING_NEEDED
Received the following measurements from the client:

• MRENCLAVE: 21a2e9dbf1491f893aae94b17ca0de7ea9592603da8eaf1c250018b30bfb6d1a
• MRSIGNER: 70199cf5824df64a4dd601416ee149d813c138afdb8c3eb8945be46b1371ec2d
• ISV_PROD_ID: 0
• ISV_SVN: 0
  [ WARNING: In reality, you would want to compare against expected values! ]
  --- [main] Re-starting myself using execvp() ---
  --- [child] Read from protected file: 'helloworld' ---
  --- [parent] Read from protected file: 'helloworld' ---

err1.txt

[dimakuv]

Hm, this information about a successful run of secret_prof_pf example doesn't help much. It just shows that your DCAP infrastructure seems to be installed correctly and works fine.

Why you see problems with the PyTorch example, I still can't understand.