add some lessons learned the hard way

README.md

@@ -43,3 +43,25 @@ kubectl create secret generic cloud-credentials --namespace velero --from-file=c
 ### Connect to services
 
 Postgres: `kubectl -n databases port-forward pod/postgres-0 5432:5432`
+
+
+# Runbook
+
+## Failing health checks
+`KUBE_CONFIG` is a secret on Drone https://drone.nocodelytics.com/nocodelytics/healthcheck/settings/org-secrets
+Value needs to come from `/etc/rancher/k3s/k3s.yaml` from the server
+
+This will expire once a year, needs to be renewed per https://docs.k3s.io/cli/certificate
+```
+# ssh into server
+systemctl stop k3s
+k3s certificate rotate
+systemctl start k3s
+```
+
+Then base64 encode it `cat /etc/rancher/k3s/k3s.yaml | base64 -i -`
+
+The same kube config, NOT encoded, goes to `~/.kube/config`, but the `server` section needs to be edited to point to the server IP
+
+## Disk space issues
+Find the persistent volume that's full, eg in `clickhouse.yaml`, edit ONLY `resources.requests.storage` section, then `kubectl apply -f ...`