How do you manage machine-to-machine differences when you have lots of different hosts?

[shawly]

I've been using chezmoi for a while now, expanding my dotfiles for all of my Linux hosts up until a few days ago. Recently I started adding my Windows dotfiles aswell and my dotfiles repo is really starting to become cluttered now and I'm kind of losing sight of what stuff gets added to which host.

I began using .chezmoidata to set some distro specific defaults for hosts and added a per-hostname config for very host specific settings. It now looks like this:

~/.local/share/chezmoi:
├── .chezmoitemplates/
│   └── machine.yaml.tmpl
├── linux/
│   ├── .chezmoidata/
│   │   ├── arch_alpine.yaml
│   │   ├── arch_debian.yaml
│   │   ├── arch_manjaro.yaml
│   │   ├── arch_ubuntu.yaml
│   │   ├── host_alpha.yaml
│   │   ├── host_beta.yaml
│   │   └── host_gamma.yaml
│   ├── .chezmoitemplates  -> ../.chezmoitemplates/    # symlink
│   └── ...   # dotfiles for linux hosts
├── windows/
│   ├── .chezmoidata/
│   │   ├── arch_windows.yaml
│   │   ├── host_omega.yaml
│   ├── .chezmoitemplates  -> ../.chezmoitemplates/    # symlink
│   └── ...   # dotfiles for windows hosts
├── .chezmoiroot    # set to windows or linux manually for my hosts
└── .gitignore

The arch_*.yaml files contain distro specific variables, like which packages shall be installed by default, the package manager used and which tools like nvm, oh-my-zsh, sdkman etc. need certain prerequisite packages installed, like this:

# yaml-language-server: $schema=../../.schema/chezmoi-machine.schema.json
---
machine:
  arch:
    ubuntu:
      package_manager:
        apt-get:
          args: '-y install'
          sudo: true
          packages:
            byobu: true
            tmux: true
            vim: true
      tools:
        p10k:
          fg_color: 202
        omz:
          package_manager:
            apt-get:
              packages:
                autojump: true
                git: true
                zsh: true
        nvm:
          package_manager:
            apt-get:
              packages:
                git: true
        sdkman:
          package_manager:
            apt-get:
              packages:
                zip: true
                unzip: true
        qemu:
          package_manager:
            apt-get:
              packages:
                dnsmasq: true
                bridge-utils: true
                dmidecode: true
                libguestfs0: true
                qemu-kvm: true
                virtinst: true
                libvirt-clients: true
                libvirt-daemon-system: true

And the host_*.yaml files contain host specific configurations:

# yaml-language-server: $schema=../../.schema/chezmoi-machine.schema.json
---
machine:
  host:
    alpha:
      headless: true
      profile: private
      type: server
      ssh_config:
        - |
          Host backups
            HostName "my-backup-space.com"
            User username-alpha
            Port 23
      tools:
        p10k:
          install: true
        qemu:
          install: true

And the machine.yaml.tmpl merges all this information for the current host like this:

---
{{ $archData := get .machine.arch "default" }}
{{ $hostData := get .machine.host "default" }}

{{- if eq .chezmoi.os "linux" -}}
{{-   if (hasKey .machine.arch .chezmoi.osRelease.id) -}}
{{-     $archData := mergeOverwrite $archData (get .machine.arch .chezmoi.osRelease.id) -}}
{{-   end -}}
{{- else if eq .chezmoi.os "windows" -}}
{{-   $archData := mergeOverwrite $archData (get .machine.arch .chezmoi.os) -}}
{{- end -}}

{{- if (hasKey .machine.host .chezmoi.hostname) -}}
{{-     $hostData := mergeOverwrite $hostData (get .machine.host .chezmoi.hostname) -}}
{{- end -}}

{{- $machineData := mergeOverwrite $archData $hostData -}}
{{ toYaml $machineData }}

Which can be included in a dotfile template with a single line in the header of a file:

{{- $machine := (fromYaml (includeTemplate "machine.yaml.tmpl" .)) -}}

This allows me to query machine specific data to ignore specific files via .chezmoiignore and only apply certain configs to specific types of hosts, like this for the .chezmoiignore:

# include p10k config
{{- if $machine.tools.p10k.install }}
!.p10k.zsh
{{ end }}

# add ssh-agent.service for non server machines
{{- if ne $machine.type "server" }}
!.config/systemd/user/
.config/systemd/user/*
!.config/systemd/user/ssh-agent.service
{{- end }}

Or I can enable p10k in my .zshrc:

{{ if $machine.tools.p10k.install }}
ZSH_THEME="powerlevel10k/powerlevel10k" 
{{ else }}
ZSH_THEME="agnoster"
{{ end }}

Where the $machine.tools.p10k.fg_color is set to orange for Ubuntu so the zsh on my Ubuntu servers is orange, while Manjaro uses the default green theme and EndeavourOS is violet.

To install the packages I use a script that leverages this setup:

#!/usr/bin/env bash

set -e

echo "[run_onchange_before_000_install-packages.sh] Start"

{{ $machine := (fromYaml (includeTemplate "machine.yaml.tmpl" .)) -}}

{{ range $package_manager, $options := $machine.package_manager }}
{{-     $packages := $options.packages -}}
{{-     $command := $package_manager -}}
{{-     $args := $options.args -}}
{{-     $sudo := $options.sudo }}

{{-     range $key, $tool := $machine.tools -}}
{{-         if (and $tool.install (hasKey $tool "package_manager") (hasKey $tool.package_manager $package_manager)) -}}
{{-             $tool_package_manager := get $tool.package_manager $package_manager -}}
{{-             if (and (hasKey $tool_package_manager "packages") ($tool_package_manager.packages)) -}}
{{-                 $packages := merge $packages $tool_package_manager.packages -}}
{{-             end -}}
{{-         end -}}
{{-     end -}}

# hash: {{ cat $packages | sha256sum }}
packages=(
{{-     range $package, $install := $packages -}}
{{-         if $install -}}
{{-             print $package " " -}}
{{-         end -}}
{{-     end -}}
)

if command -v {{ $command }} &>/dev/null; then
    echo "Installing packages via {{ $command }}: ${packages[@]}..."
    {{ if $sudo }}sudo {{ end }}{{ $command }} {{ $args }} "${packages[@]}" || exitCode="$?"
    if [[ "${exitCode:-0}" -ne 0 ]]; then
        echo "Failed to install packages, aborting!"
        exit "${exitCode}"
    fi
else
    echo "Package manager {{ $command }} not found, skipping package installation!"
fi

{{ end }}

echo "[run_onchange_before_000_install-packages.sh] End"

With this setup I can set up a new host, add a yaml for it to my .chezmoidata configs and initialize the host with my dotfiles via chezmoi init easily. But I am not yet sure if stuff like installing packages etc. should be done through chezmoi. I'd rather use Ansible for managing the host itself. While I am satisfied with this setup, I think it is not perfect and I need some inspiration on how I could improve this.

How do you manage lots of machines? Are you only using templates or do you have a more generic approach?

[Lockszmith-GH]

I don't know if it will unclutter it for you, but I've been using a layered approach using multiple chezmoi sources and/or repos for situations of "some-common-some-different".

When there is too much divergance, then I would prefer a separate repo, yet I still want the convinience of chezmoi.

The cost however, is to CONTEXT switch between the differnt sources.
If you feel you can do that, then check out my post about it #2574 . And if you are intereseted in specifics, feel free to ask questions (either here or in the post).

[shawly]

Thanks! I've already looked at your post and it is not exactly what I'm looking for, even though I find the idea really good!

I think the biggest gripe I have with my setup is that it does more than I expect from a dotfiles repo, which is installing packages.
And I think I will change this by only validating if necessary packages are installed, instead of just installing them. Then I can use Ansible for host management. This will simplify my dotfiles setup and keep its purpose singular, which is configuring my main user on my systems.

Though I am still interested how people manage lots of machines and still keep their repo maintainable and clean!

[twpayne]

Note that you can call Ansible (or Puppet or Chef or Salt or whatever) from a chezmoi run_ script. If you want to do things the other way around (i.e. call chezmoi from Ansible) there's https://github.com/hussainweb/ansible-role-chezmoi and other related software.

[Lockszmith-GH]

I've been eyeing both intall.doctor and a project named comtrya for this. No experience yet.