K3s PaaS

• Documentation
• Original tutorial (FR)

Compatibility Matrix :

OS	Status
Darwin	OK
Linux	NO

New Nix system (beta)

Setup (Darwin)

Nix installation :

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

echo '. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.fish' >> ~/.config/fish/config.fish

Build

Set up nixos-darwin :

make bootstrap

Note : make bootstrap-x86 for cnotabo build and deployment.

Build an image between available nixos configuration contabo and qcow2 :

Supported systems are aarch64-linux, x86_64-linux, aarch64-darwin and x86_64-darwin.

nix build .#nixosConfigurations.default --system x86_64-linux

Uninstall on Darwin:

When builder environment not starting (no libvirtd.sock)

./result/sw/bin/darwin-uninstaller

Terraform project secrets

You have to create a secrets file utilising following sops command :

sops secrets/prod.yaml

Note : sops follow environment variables from default flake shell to encrypt files with the default id_ed25519 ssh key.

1. Contabo (VPS)

contabo_credentials :

contabo_credentials:
  oauth2_client_id: "client-id"
  oauth2_client_secret: "secret"
  oauth2_pass: "password!"
  oauth2_user: "mail@mail"

Seek for credentials in API

contabo_instance :

cntb config set-credentials --oauth2-clientid id --oauth2-client-secret secret --oauth2-password "contabo-dashboard-pass"
cntb get instances

2. Gandi (domain)

• paas_base_domain : Order a domain on gandi
• gandi_token : Generate a Personal Access Token on gandi organization

Warn : Delete @ record for your domain on gandi

3. Tailscale (SSH VPN)

tailscale_oauth_client : - id : Register on tailscale and get key on admin console - secret : retrieve it during step above.

tailscale_trusted_device : Approve your device on tailscale with tailscale login and recover its tailscale hostname.

4. Github (Authentication & users)

github_token : https://github.com/settings/tokens and create a token with scopes repo, user and admin. github_client_id : Create a new OAuth App. github_client_secret : On new OAuth App ask for a new client secret. github_organization : : Your github organization name. github_team : : Your github team id.

5. Cert-manager (TLS)

cert_manager_email : a valid email to register on letsencrypt.

Terragrunt apply

Cloud (contabo)

make terragrunt/cloud/contabo

Network (tailscale)

make terragrunt/network/contabo

infra (k8s)

make terragrunt/paas/contabo

Note : Retrieve any config with :

make terragrunt/network/contabo TF_CMD='output -json k3s_config | yq -p json -o yaml'

Install in local

Here is the command to set up the paas on a libvirt vm.

make terragrunt/cloud/local
make terragrunt/network/local
make terragrunt/paas/local

Then you have to do make trust-ca to trust paas internal CA on your system.

Cheat Sheet

Nix

See derivations of a build :

nix derivation show -r '.#nixosConfigurations.aarch64-darwin.default'

Filter derivations by name :

nix derivation show -r '.#nixosConfigurations.aarch64-darwin.default' | jq -r '.[] | select(.name | contains("cert-manager"))'

Debug flake :

nix --extra-experimental-features repl-flake repl '.#'

Free unused derivations :

nix-store --optimise

Repair nix store :

nix-store --verify --check-contents --repair

Libvirt

Undefine pool :

virsh -c qemu:///system pool-undefine libvirt-pool-k3s-paas

Undefine vm to avoid conflicts :

virsh -c qemu:///system undefine --nvram localhost-0

Open console :

virsh -c qemu:///system console localhost-0

Exit with Ctrl + + or Ctrl + ] on linux.

See this SO thread if you keep struggling.

Openssl

Generate a sha512crypt password :

openssl passwd -salt zizou -6 zizou420!

Kubectl

Set context :

kubectl config set-cluster default --server=http://k3s-paas-master-0:6443
kubectl config default test-cluster

See all pods :

kubectl get po -A

See any assets :

kubectl get all -A

Release

git tag nixos-stable -f && gp --tags --force

Contabo

Retrieve images :

cntb get images --imageType custom

Import existing image in terraform :

terraform import module.contabo.contabo_image.k3s_paas_master_image uuid

Trouble shoot

Permissions errors on result directory in Terragrunt

chmod -fR 755 terragrunt/**/.terragrunt-cache/ && chmod -fR 755 result || true

Dns Cache

On macOS, dnsmasq starts in background, you might need to force a refresh of the dns cache :

sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Cilium secrets namespace stuck in Terminating

kubectl delete apiservice v1beta1.metrics.k8s.io