nocodelytics
/
infrastructure
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

add some lessons learned the hard way

This commit is contained in:
Miroslav Batchkarov 2024-12-06 13:00:14 +01:00
parent fb72b3c67f
commit c772633e72
1 changed files with 22 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
README.md
View File

@ -43,3 +43,25 @@ kubectl create secret generic cloud-credentials --namespace velero --from-file=c
### Connect to services
Postgres: `kubectl -n databases port-forward pod/postgres-0 5432:5432`
# Runbook
## Failing health checks
`KUBE_CONFIG` is a secret on Drone https://drone.nocodelytics.com/nocodelytics/healthcheck/settings/org-secrets
Value needs to come from `/etc/rancher/k3s/k3s.yaml` from the server
This will expire once a year, needs to be renewed per https://docs.k3s.io/cli/certificate
```
# ssh into server
systemctl stop k3s
k3s certificate rotate
systemctl start k3s
```
Then base64 encode it `cat /etc/rancher/k3s/k3s.yaml | base64 -i -`
The same kube config, NOT encoded, goes to `~/.kube/config`, but the `server` section needs to be edited to point to the server IP
## Disk space issues
Find the persistent volume that's full, eg in `clickhouse.yaml`, edit ONLY `resources.requests.storage` section, then `kubectl apply -f ...`