Monitoring your Hetzner Cloud Kubernetes Cluster

[boxedcode]

Hi there,

First of all, a general question for the community: What setup do you guys have for monitoring your Kubernetes cluster? The popular option is Prometheus/Grafana, but wondering if there are other options you guys use in practice.

Secondly, much like how in your documentation, you have the following examples:

• Useful Cilium commands
• Ingress with TLS
• etc.

Do you have an example of how to set up monitoring with Prometheus/Grafana for this module? I can set it up directly against Kubernetes, but I want to do it the recommended way for this module so that when I someday decide to teardown my infrastructure using terraform, it'll also tear down the monitoring setup as well.

Thanks,

[aleksasiriski]

[ifeulner]

Easiest way: Install rancher management console, and deploy Prometheus / Grafana / Loki (Logs) from the app section.

Then set your alarms etc. accordingly.

[mysticaltech]

@aleksasiriski Like @ifeulner said, at some point, features like monitoring are better installed via Rancher, and we support Rancher.

So @boxedcode, just follow @ifeulner's advice, deploy Rancher on your cluster (or another Single node cluster, which needs to be cx21 minimum, I do that personally), and use it to manage your cluster(s) and deploy things like monitoring in just a few clicks!

[boxedcode]

Thanks everyone, appreciate the help. I'll go down the route of rancher on a separate host.

[gefloh]

@aleksasiriski Like @ifeulner said, at some point, features like monitoring are better installed via Rancher, and we support Rancher.

So @boxedcode, just follow @ifeulner's advice, deploy Rancher on your cluster (or another Single node cluster, which needs to be cx21 minimum, I do that personally), and use it to manage your cluster(s) and deploy things like monitoring in just a few clicks!

Could it be that this doesn't work with arm64 clusters @mysticaltech? I played around a bit with Rancher but I don't see the monitoring in the catalog on my arm64 cluster. On the Rancher single-node CX21 cluster itself, it is in the catalog.

[mysticaltech]

@gefloh It could be! See https://ranchermanager.docs.rancher.com/v2.5/getting-started/installation-and-upgrade/advanced-options/enable-experimental-features/rancher-on-arm64

[mysticaltech]

@aleksasiriski Like @ifeulner said, at some point, features like monitoring are better installed via Rancher, and we support Rancher.

So @boxedcode, just follow @ifeulner's advice, deploy Rancher on your cluster (or another Single node cluster, which needs to be cx21 minimum, I do that personally), and use it to manage your cluster(s) and deploy things like monitoring in just a few clicks!

[thisIsLoading]

just tried that.

Error: INSTALLATION FAILED: chart requires kubeVersion: < 1.28.0-0 which is incompatible with Kubernetes v1.28.4+k3s2

looks like we're just too new for rancher :/

[aleksasiriski]

You could easily downgrade to v1.27, just set it in kube.tf and run terraform apply

[aleksasiriski]

My suggestion nowadays would be to deploy SigNoz instead and use that for monitoring. It supports logs, metrics and traces, all in one dashboard. It's pretty straightforward to deploy using the official Helm chart and it automatically monitors the whole K8s cluster (nodes, pods and everything). It's also super easy to make it monitor Prometheus exported metrics (for example Authelia can export metrics on /metrics path and 9959 port, and you only need to add the following to the pod spec template:

annotations:
  signoz.io/scrape: "true"
  signoz.io/port: "9959"
  signoz.io/path: /metrics
  apm.signoz.io/scrape: "true"
  apm.signoz.io/port: "9959"
  apm.signoz.io/path: /metrics

For Hetzner you can use these helm values:

global:
  cloud: hcloud

[dustinmoris]

Can you share more information on how you set up k3s with SigNoz? I installed it using the Helm chart and didn't see any metrics in SigNoz from my k3s cluster. Also which storage class did you configure SigNoz with? Are you just using local-path or a Hetzner volume? Thank you!

[aleksasiriski]

I can give you my Helm values file for the SigNoz, but it's mostly the defaults with some resource requests and limits. Do you have K3s installed with metrics server?

[dustinmoris]

I should have metrics server installed, I actually set enable_metrics_server = true explicitly even though true is the default value anyway. Sharing your SigNoz values.yaml would be appreciated. What is the expected outcome of the default SigNoz installation on my k3s cluster in Hetzner? I thought importing the host metrics dashboard and the k8s-infra-metrics dashboard should show me data after the default install?

[dustinmoris]

I worked it out myself. If anyone comes here wondering the same as I did, SigNoz changed the Helm chart and it doesn't include the k8s-infra chart anymore, so this one needs to be installed additionally to the main chart. You also have to configure the Otel collector service address when installing the chart. This is what I did:

helm repo add signoz https://charts.signoz.io --force-update

helm install signoz signoz/signoz -n monitoring --create-namespace -f values.yaml

helm install signoz-k8s-infra signoz/k8s-infra \
  -n monitoring \
  -f values.yaml \
  --set otelCollectorEndpoint=signoz-otel-collector:4317

If you name your release differently then you have to change the endpoint name as well.

My values.yaml is super short:

global:
  storageClass: local-path
  cloud: other
  clusterName: k3s

I explicitly set local-path as the storageClass, because I want to ensure that all db writes happen on the local node for speed. If you want HA you can modify the SigNoz chart to scale the ClickHouse cluster, but in this instance I didn't need it because I only need a short retention of logs, metrics and traces anyway and if the entire cluster crashes then I don't care if the SigNoz data is lost because it's only relevant in the moment and not years later.

I also configured cloud: other rather than cloud: hcloud because I saw that if you specify cloud: hcloud SigNoz might still attempt to create a Hetzner volume for its data, which as I said I definitely don't want to. This needs to be fast, one of the main reasons SigNoz is using ClickHouse is for its speed as a column based store. The cloud setting has no other implication on the deployment other than the volume selection, so other was perfectly fine, to keep it unopinionated and solely rely on my configured storageClass for the PVCs.