humble-jungle — private infrastructure lab

A private cloud in Switzerland, designed, built, and run solo by Stanisa Obradovic (platform engineer). The stack is deliberately ordinary — Proxmox VMs, kubeadm Kubernetes, Cilium, Argo CD, sealed-secrets, self-hosted models on a 128 GB unified-memory GPU node. The engineering worth reading is what broke once it ran like a real system:

Placement fought GitOps. The first version of sitepilot — a Go operator for site-aware workload placement — enforced placement by patching Deployments. Argo CD owns those objects: two writers on one field is a drift war, plus a rollout on every round. An adversarial review caught it before it shipped; the redesign injects placement at Pod admission instead, and the Deployment is never touched. The field note publishes the actual controller function and where the design stops short.

Cloud models are an egress path. Every prompt bound for an external model is scanned by a pre-call guardrail in the model gateway — pattern plus entropy, benchmarked against a downloadable 38-case corpus: 17/17 secret shapes caught, 0/21 false positives, known escapes documented. A guardrail, not a DLP product; local models are exempt because their data never leaves the node.

The backup that wasn't. The GitOps repo looked fully reproducible — until writing the disaster-recovery runbook proved it wasn't: the sealed-secrets private key lived only inside the cluster it was meant to resurrect. A near-miss found before any outage, fixed, and now exercised by a drill — 15/15 SealedSecrets recovered using only the offline key backup.

A scheduled agent works around the clock on read-only credentials: it can look and propose; a human merges. Git stays the deployment boundary.

The numbers on this page are live — a job inside the cluster pushes a sanitized, ed25519-signed snapshot every 15 minutes, strictly outbound. Verify without a browser: curl -L humble-jungle.com returns a live status board; /api/status serves the JSON and /api/status/signed the signature.

The full site is a JavaScript app: live dashboards, interactive demos, and ten field notes — the decisions, the rejected designs, the measurements, the post-mortem.

Operator: shouldwe.link