Blog on Statefulops

10: day 2 ops

Tue, 26 May 2026 08:20:10 +0000

Building a cluster is the easy part. Operating one for two years without losing data is the actual job. This last post in the series covers the three things that distinguish a cluster you can rely on from a cluster you’ve built: the upgrade workflow, etcd backups, and the failure modes you’ll meet in real use.

9: storage ingress

Tue, 26 May 2026 08:20:09 +0000

The cluster is healthy and HA, but it’s not yet useful. Stateful workloads need somewhere to put their data, and HTTP workloads need a way to be reachable. This post installs local-path-provisioner as a default StorageClass, ingress-nginx as our ingress controller, and cert-manager so we can pull Let’s Encrypt certificates without thinking about it.

8: ha kube vip

Tue, 26 May 2026 08:20:08 +0000

One control plane node is enough for a working cluster. It is not enough for an HA cluster. If cp-1 goes down right now, every kubectl call hangs, no pods can be scheduled, and worker kubelets eventually start panicking about node leases they can’t renew. This post adds cp-2 and cp-3 to fix all of that.

7: workers

Tue, 26 May 2026 08:20:07 +0000

One control plane node, one CNI, zero workers. Time to fix the third one. By the end of this post we’ll have worker-1 and worker-2 joined, Cilium running on them, and three nodes Ready. The mechanics are simple; the failure modes are where the interesting bits live.

6: Cillium

Tue, 26 May 2026 08:20:06 +0000

At the end of post 5 we had a control plane running but cp-1 stuck in NotReady. That’s because kubelet refuses to be Ready until a CNI plugin has installed itself into /etc/cni/net.d/ and can hand out pod IPs. This post installs Cilium, gets the node Ready, and gives us a working pod network with kube-proxy already replaced.

5: Kubeadm init

Tue, 26 May 2026 08:20:05 +0000

This is the post where we actually get a Kubernetes API server running. After this, you’ll have a single-node control plane on cp-1 that’s already wearing its kube-vip VIP — ready for Cilium in post 6, workers in post 7, and the other two control plane nodes in post 8. The cluster won’t be highly available yet, but the certificates, the API endpoint, and the etcd configuration will all be HA-ready from the moment of birth.

4: Install kubeadm

Tue, 26 May 2026 08:20:04 +0000

Three binaries, one apt repo, two gotchas. This is the shortest post in the series and one of the most consequential to get right, because the choice you make about which Kubernetes minor version to install in this step is the one you’ll live with until you decide to upgrade.

3: Containerd

Tue, 26 May 2026 08:20:03 +0000

Kubelet doesn’t run containers directly. It talks over a socket to a container runtime, using the Container Runtime Interface (CRI). The runtime does the actual work — pulling images, creating namespaces and cgroups, calling out to runc to start the process, and reporting back. containerd is the runtime kubelet most commonly talks to these days, and it’s what we’re going to use.

2: VM Prep

Tue, 26 May 2026 08:20:02 +0000

If post 1 was the setup, this post is the part where we make sure the VMs are actually ready to host a Kubernetes cluster. Nothing in this post is glamorous. All of it is the reason your cluster either comes up cleanly when you run kubeadm init, or doesn’t.

1: Why kubeadm and what are we building

Tue, 26 May 2026 08:20:01 +0000

Most “install Kubernetes” tutorials fall into one of two camps. There’s the five-minute k3s/kind crowd, where you run a single command and end up with something that boots but doesn’t behave like the cluster you’ll meet at work. And there’s the production-architecture-from-day-one crowd, where post one is a 4,000-word essay on etcd quorum before you’ve typed kubeadm.

Container Scanning on GitLab MRs with Trivy and Grype

Sun, 12 Apr 2026 17:53:26 +0000

In the previous post we got Renovate running on self-hosted GitLab CE, automatically discovering repos and raising MRs when new image versions are available. That handles the update workflow — but it doesn’t tell you whether the new image is actually safe to run. This post adds container scanning to the picture: every MR that touches a Docker Compose file gets scanned by both Trivy and Grype before it can merge.

Automated Dependency Updates on Self-Hosted GitLab with Renovate

Sun, 12 Apr 2026 09:32:39 +0000

If you’re running a self-hosted GitLab CE instance with a bunch of repos, keeping dependencies and Docker image tags up to date across all of them is a pain. Renovate solves this — it’s an open-source tool that opens MRs automatically when new versions are available. This post walks through setting it up from scratch on GitLab CE, including autodiscovery across a whole group and a 7-day hold before any MR gets raised.

Everything on StatefulOps — where to start

Wed, 25 Mar 2026 08:04:57 +0000

If you’ve landed here from a search or a shared link and you’re not sure where to start — this is the post for you.

StatefulOps covers the stuff that actually matters when you’re running production Kubernetes: getting the fundamentals solid, wiring up GitOps so deployments stop being scary, and building the observability layer so you know what’s happening at 2am when things go sideways. Three series, 27 posts, all written for engineers who’d rather read something honest than wade through another docs-flavoured tutorial.

Part 09 — Observability in practice: debugging real incidents

Mon, 23 Mar 2026 08:32:04 +0000

Everything in this series has been building to this. You have Prometheus scraping metrics, Loki collecting logs, Tempo storing traces, and Grafana connecting them. You have dashboards, alerts, and runbooks. Now let’s use them.

This post works through three real incident scenarios — the kind that happen in actual Kubernetes clusters, not contrived tutorial examples. Each one demonstrates the investigation workflow from first alert to root cause, using the actual queries you’d run. The scenarios get progressively more complex: a latency spike with no obvious cause, a slow memory leak leading to an OOMKill, and a cascading failure across multiple services.

Part 08 — The Grafana stack: putting it all together

Mon, 23 Mar 2026 08:29:51 +0000

The last six posts have covered the three pillars individually — Prometheus for metrics, Loki for logs, Tempo for traces. Each one is useful on its own. But the real leverage comes from using them together, connected in a way that lets you move fluidly between signals during an investigation without losing context.

Part 07 — Traces: following a request across services

Mon, 23 Mar 2026 08:28:40 +0000

Here’s the scenario that distributed tracing exists to solve. A user reports that checkout is slow — sometimes five seconds, sometimes twelve. Your error rate is fine. Your metrics look normal. CPU is healthy. Memory is healthy. Everything on every dashboard is green except that users are clearly experiencing something bad.

Part 06 — Logs: collecting, shipping, and querying

Mon, 23 Mar 2026 08:24:26 +0000

Metrics tell you something is wrong. Alerts wake you up about it. Then you need to understand what actually happened — and for that, you almost always end up in logs.

Logs are the detailed record of what your application was doing at the time things went wrong. The error message. The stack trace. The specific request that triggered the failure. The sequence of events that led up to it. Metrics can tell you that error rate spiked at 14:32. Logs can tell you that at 14:32:07, pod payments-api-6d8f9-xk2p started throwing connection refused errors trying to reach fraud-detection-svc:8080, and here are the request IDs of the forty-three affected transactions.

Part 05 — Alerting: being woken up for the right reasons

Sun, 22 Mar 2026 17:02:26 +0000

There’s a reliable way to tell if a team’s alerting is working well. Ask them how they feel about their on-call rotation. If the answer involves words like “exhausting,” “constant noise,” or “I stopped looking at alerts because most of them are false alarms” — the alerting is broken. Not broken in the sense of technically malfunctioning. Broken in the sense that it’s no longer doing its job.

Part 04 — Grafana: turning metrics into understanding

Sun, 22 Mar 2026 17:00:20 +0000

A common failure mode in Kubernetes observability goes like this. The team installs kube-prometheus-stack. They get Grafana. They open the pre-built dashboards and see hundreds of panels showing CPU, memory, network, disk, pod counts, container restarts, API server latency, etcd commit duration, and thirty other things. They feel like they have visibility. They add more panels for their own applications. The dashboard grows. It looks impressive on a monitor in the office.

Part 03 — PromQL: querying your metrics

Sun, 22 Mar 2026 16:59:08 +0000

Prometheus without PromQL is a database you can’t read. The metrics are there, the time series are accumulating, but until you can query them meaningfully you’re just storing numbers. PromQL is what turns those numbers into answers.

It has a reputation for being difficult. That reputation is partly deserved — PromQL has some genuinely unintuitive moments — but it’s mostly unearned. The core concepts fit on one page, and once they click, you can write useful queries almost immediately. The complexity people run into is usually from jumping to advanced expressions before understanding the fundamentals.

Part 02 — Metrics: Prometheus from scratch

Sun, 22 Mar 2026 16:56:55 +0000

If you’ve spent any time around Kubernetes, you’ve heard of Prometheus. It shows up in helm charts, in monitoring stack recommendations, in job descriptions. It’s become so embedded in the Kubernetes ecosystem that it can be easy to assume you should just install it and move on without understanding what it actually does.

Part 01 — Why observability is different from monitoring

Sun, 22 Mar 2026 16:55:26 +0000

Here’s a scenario. It’s a Tuesday afternoon and your on-call phone goes off. The alert says “RabbitMQ queue depth exceeds 10,000 messages.” You look at your dashboard. Queue depth is definitely high. Memory usage looks elevated. CPU is fine. The alert doesn’t tell you why the queue is backing up — just that it is. You spend the next two hours digging through logs, checking deployment history, and asking colleagues if anything changed recently. Eventually you find it: a downstream service started timing out six hours ago, which caused consumers to slow down, which caused the queue to fill. The alert fired too late. The dashboard told you what but not why. And even finding the “why” required piecing together signals from three different places by hand.

Part 8 — Running GitOps at scale: multi-cluster and multi-team

Fri, 20 Mar 2026 16:02:04 +0000

Everything in this series so far has implicitly assumed a relatively simple setup: one team, one or a handful of clusters, one ArgoCD instance, a config repo with a reasonable number of applications. That setup works well and scales further than you’d think.

Part 7 — Progressive delivery: beyond all-at-once deploys

Fri, 20 Mar 2026 15:57:12 +0000

Rolling updates, which we covered back in the Kubernetes series, are a significant improvement over taking your service down to deploy a new version. But they have a blind spot: the moment the rollout completes, 100% of your traffic is hitting the new version. If that version has a subtle bug — one that only manifests under real production load, with real user data, at real traffic volumes — you find out the hard way.

Part 6 — Secrets in a GitOps workflow

Fri, 20 Mar 2026 15:53:20 +0000

Everything we’ve covered so far has a satisfying elegance to it. Desired state in Git. ArgoCD watching. Cluster reconciling. The whole system is auditable, reproducible, and self-healing.

Then someone asks: “where do the database passwords go?”

And the answer can’t be “in Git.” Not in plain text, anyway. Git repositories — even private ones — get cloned to developer laptops, backed up to third-party services, included in contractor access grants, and occasionally leaked. Committing a production database password to Git is not a theoretical risk; it’s a when-not-if.

Part 5 — Helm and Kustomize in a GitOps world

Fri, 20 Mar 2026 15:47:09 +0000

By now you have a Git repo, ArgoCD watching it, and a folder structure that separates environments. The next problem is one that every team hits within the first week: your payments API in dev and your payments API in production are almost identical, but not quite. Different image tags, different replica counts, different resource limits, different database hostnames.

Part 4 — Structuring your Git repository

Fri, 20 Mar 2026 15:42:08 +0000

Here’s a conversation that happens on almost every team adopting GitOps for the first time. You’ve got ArgoCD running, you’ve written your first Application, you’ve seen the sync work. Now someone asks: “where exactly does all this YAML live?”

It’s a deceptively important question. Get the repo structure wrong early and you’ll spend months untangling it. Get it right and it scales from one service to hundreds without much additional ceremony.

Part 3 — ArgoCD: your first GitOps controller

Fri, 20 Mar 2026 15:28:38 +0000

The last two posts covered the theory. GitOps is a good idea. The four principles make sense. Now let’s talk about the software that actually implements it.

ArgoCD is the most widely used GitOps controller for Kubernetes. It watches a Git repository, compares what’s there to what’s running in your cluster, and continuously reconciles the two. It also comes with a genuinely useful web UI — which, in a world of Kubernetes tooling that’s often CLI-only, is a pleasant surprise.

Part 2 — The four principles of GitOps

Fri, 20 Mar 2026 15:25:07 +0000

GitOps gets thrown around a lot as a buzzword, which means it gets applied to a lot of things that aren’t quite it. “We store our manifests in Git” — not quite. “Our CI pipeline runs kubectl apply” — closer, but still no. “We use ArgoCD” — probably, but not automatically.

Part 1 — What is GitOps and why does it exist?

Fri, 20 Mar 2026 15:21:56 +0000

If you’ve been running Kubernetes for a while, you’ve probably settled into a deployment workflow that looks something like this. A developer finishes a feature. CI builds and pushes a new image. Someone — maybe a script, maybe a human — runs kubectl apply against the cluster. The new pods roll out. Everyone moves on.