Design notes and TODOs

[cjellick]

Just capturing the design notes and TODOs from our design sessions with @ibuildthecloud @dweomer. These notes are incredibly raw. Just transfering my scribblings, so take them with a grain of salt

Background on architecture and the upstream components we are stitching together

• unanswered design question - embed or separate processes for containerd and buildkit
• Nice feature to keep in mind: docker for desktop shares images with k8s
• The buildkit code has pretty confusing to work with. Has more features than what we assume to be its core functionality. Might be used elsewhere in the docker suite or projects/products
• Can CRI support push and tag> It doesn't today but maybe we can work with the CRI team to see if CRI could be expanded to include that
• The bulidkit currently in k3c has some patches for bugs. They need upstreamed. This could be a quick/easy win as far as k3c work goes
• CRI is patched to a custom build for the following reasons:
  • It doesn't have a flow that supports creating, starting, and attaching to a container such that you can be sure you never miss a byte of the output. We can maintain this but should try to work with CRI team to see if it can be improved upstream.
  • push behavior - need a resolver. needd to share memory context to get at the resolveer. exposed the method in CRI server to get the resolve. Would be nice to have the upstream version properly expose this. Maybe even as just a library call?
• containerd version we use also carries patches, generally to get fixes more quickly than the upstream

Functionality Gaps

• DNS - two pods foo and bar cant resolve each other as foo and bar. solve by adding in a coredns service, but just add a resolver to the pod, do not manage /etc/hosts
• Rootless

K3s integration

• why k3s and containerd are 1 binary but they are two processes
• could swap k3c in for contianerd (see daemon: should be full drop-in replacement for containerd #49)
• weirder bigger : If you create a container via k3c, the k3s kubelet will recognize and kill it because they are all in the same k8s.io containerd namespace. Possible solutions:
  • k3c run creates a static pod
  • use a different containerd namespace for k3c commandline? Downside is that k3c ps wont list k3s pods, which is kind of the point for making debugging better
• patch CRI to leverage "unlisted" gRPC header fb7c31e
• k3s->k3c symlink or expose commands directly? (i dont recall the context of this note)

RancherOS feature parity

• Static pod support in cloud-config
• Why k3os isnt a good ros replacement (yet)
  • no good debugging suite
  • cloud-config compose syntax

k3c can address the debug-ability and simply supporting existing k8s static pod syntax in cloud-config can be enough to replace the compose syntax

TODOs/action items

Buildkit:

• find a better embedding approach
• cleanup and upstream bug fixes

CRI:

• cleaner and upstream-friendly approach for sharing resolver (see cri: upstream resolver hack #51)
  containerd/CRI
• figure out the containerd namespace overlap issue (that would cause the kubelet to destroy containers created via k3c run) (see fb7c31e)
  Fix CI
• drone setup - amd64 - no dapper.
• Use GH actions if possible. Might need our own runner
• Take a look at what Erik did for k3s.
• look at the GH model for keeping secrets out of the build
• ARM? Arm and Arm64 runners?