From c772633e72709ca85cb8789104b7e9b0bc69be16 Mon Sep 17 00:00:00 2001
From: Miroslav Batchkarov <mbatchkarov@gmail.com>
Date: Fri, 6 Dec 2024 13:00:14 +0100
Subject: [PATCH] add some lessons learned the hard way

---
 README.md | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/README.md b/README.md
index 173770b..d8577d3 100644
--- a/README.md
+++ b/README.md
@@ -43,3 +43,25 @@ kubectl create secret generic cloud-credentials --namespace velero --from-file=c
 ### Connect to services
 
 Postgres: `kubectl -n databases port-forward pod/postgres-0 5432:5432`
+
+
+# Runbook
+
+## Failing health checks
+`KUBE_CONFIG` is a secret on Drone https://drone.nocodelytics.com/nocodelytics/healthcheck/settings/org-secrets
+Value needs to come from `/etc/rancher/k3s/k3s.yaml` from the server
+
+This will expire once a year, needs to be renewed per https://docs.k3s.io/cli/certificate
+```
+# ssh into server
+systemctl stop k3s
+k3s certificate rotate
+systemctl start k3s
+```
+
+Then base64 encode it `cat /etc/rancher/k3s/k3s.yaml | base64 -i -`
+
+The same kube config, NOT encoded, goes to `~/.kube/config`, but the `server` section needs to be edited to point to the server IP
+
+## Disk space issues
+Find the persistent volume that's full, eg in `clickhouse.yaml`, edit ONLY `resources.requests.storage` section, then `kubectl apply -f ...`
\ No newline at end of file