Better secret support for PAT, LLM keys, etc

[krassowski]

Feature description

Currently any secrets passed down to the single-user Jupyter server via configuration or environment variables can be discovered by the user. This is usually not a problem if these tokens are well scoped, or in case of model keys - have appropriate billing limits/controls. This is not always possible.

Value and/or benefit

• Ability to confidently share example private repositories with the users with guarantee that they will not be able to clone the repository without tracking outside of the platform (jupyterlab-gallery use case)
• Reduced risk of the LLM keys leaking (jupyter-ai use case)

Anything else?

This is a follow-up to #2501.

Possible solutions

• a) run certain server extensions in a separate container with additional secret mounts; proxy the API to the main container
• b) change the file system permissions of the config files so that users cannot see them; start the single-user juypter server with elevated permissions and then drop down to the user with lower permissions (see Implement support for jupyterlab-gallery config #2501 (comment)).

I would need to explore it a bit further but while technically feasible, (b) appears not worth the trouble because changing the user identity via SETGID may be insecure.

[Adam-D-Lewis]

I agree this would be great. For context, we have considered Vault and SOPS for secret management in the past with Vault appearing to be the most promising if I remember correctly.

[viniciusdc]

Thanks for opening the issue, @krassowski. I think, your original intention with this issue was to target the use cases of the new extension and their access tokens present as env vars right? (due to them being serviced?). Though, as @Adam-D-Lewis pointed out above, we also have more general solution to the overall secret management conversation, that could be interesting as a secodna point of view.

In any case, I think (b) would've been the best approach. However, besides your appointed problem, there are many indirect dependencies in the user and system permissions within the user pod itself that other dependent services (such as conda-store or dask) could interact with, leading to possible issues later on, so I too, would like to avoid touching it if possible.

I also like the first option, (a), as it seems more manageable at first glance and, at least to me, follows a mico-segmentation workflow standard. I would like to know more on what are the requirements to get into that if possible, as my perspective on this matter is heavily oriented to k8s itself.

[krassowski]

Something potentially related: https://github.com/danilopeixoto/jupyterhub-ai-gateway

[krassowski]

I managed to add a second (sidecar) container with additional access to secret mounts in the user pod (option a). It's working 🎉

For now it runs the same jupyterhub singleuser server. Because it is on the same network and with the same token, user is able to access it with jupyter-server-proxy. Now there is a couple of more things to take care of

• (I) make sure that we block the access to contents api and anything not needed in the second container
• (II) reduce the cost of the second container:
  • consider running juypterlab-gallery as an individual app rather than an entire server (it should be already possible, just needs testing); this would also help with (I); this will require a bit more work to make this work with JHub auth though
  • add resource limitations
  • try using native sidecar containers again (I had some trouble with k8s here, it is still in beta though so maybe later)
  • consider using a dedicated image rather than re-using the main image.
• (III) consider URL rewriting to avoid the need to configure a different endpoint on jupyterlab-gallery (and make it also possible to work with jupyter-ai, etc)
• (IV) solve a couple issues with how dependencies are passed around in profile creation (and in configmaps)

[krassowski]

Because jupyter-gallery would now run in a separate sidecar container we need to redirect API calls. There are two solutions:

• (1) tell jupyter-gallery to use another endpoint, by adding overrides.json:

   "jupyterlab-gallery:plugin": {
       "endpoint": "proxy/${var.gallery-sidecar-port}/jupyterlab-gallery"
   },

• (2) add traefik middleware rewriting the URL transparently, like:

  redirectRegex = {
       regex       = "^https://${var.external-url}/user/([^/]+)/jupyterlab-gallery(/?[^/]+)$"
       replacement = "https://${var.external-url}/user/$${1}/proxy/${var.gallery-sidecar-port}/jupyterlab-gallery$${2}/"
       permanent   = true
     }

I lean slightly towards the latter to hide the implementation detail from the app and avoid having to display settings for jupyterlab-gallery plugin in JupyterLab UI, but I wanted to check with the team here to confirm that there are no objections.

Another advantage of (2) option is that we can also add a rule blocking any other traffic to proxy on the chosen port. The downside is that it will be a tad harder to debug when things go wrong - if someone was to remove this rule, the extension would appear to be partially working because it would be hitting the endpoint in the main container (although it would not be getting any config).

[krassowski]

Also, with (2) we could apply the same to extensions which we do not control (like jupyter-ai) where we cannot change the endpoint easily.

[krassowski]

While running the full server app in a sidecar is easy, it is not currently possible to make it load only one selected extension (here GalleryApp).

• (a) jupyter-server has an override for disabling extensions but not a block-list Allow-list for server extensions? jupyter-server/jupyter_server#1445
• (b) jupyterhub does not allow to swap the ServerApp for an ExtensionApp (not without loosing the auth and heartbeat customization) Support running jupyter-server ExtensionApp as standalone servers jupyterhub/jupyterhub#4857

Until we hear from upstream on whether they would accept a patch I think the best way forward is to run the GalleryApp as a server extension without JupyterHub customisations, but long term we would ideally want to revisit it.

[krassowski]

I had quite some success with applying the JupyterHub patches to jupyter-server spawned by gallery in standalone mode. The one last thing which does not work is the progress bar as proxying of SSE is not yet supported by jupyter-server-proxy but there is an active PR for this (jupyterhub/jupyter-server-proxy#479).

[aktech]

(2) add traefik middleware rewriting the URL transparently, like:

This sounds like a better approach to me as well.

[kcpevey]

Status update:
(a) Blocked on #2602
(b) Blocked on jupyterhub/jupyter-server-proxy#479