Desired State vs Actual State: The Kubernetes Paradigm¶

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

The entire Kubernetes architecture is built on one idea. You say "I want 3 replicas of a web server." The system reads this as the desired state and constantly corrects the actual state to match it. This is the declarative approach.

Imperative vs Declarative¶

Imperative (like a script): "create container A, then B, then C." If B crashes, the script doesn't know what to do.

Declarative (like Kubernetes): "there should be 3 containers." If one crashes, the system sees "there are 2, need 3" and creates another one.

graph TD
    subgraph "Imperative"
        I1["Step 1: create A"] --> I2["Step 2: create B"]
        I2 --> I3["Step 3: create C"]
        I3 --> I4["B crashed... now what?"]
    end

    subgraph "Declarative"
        D1["Desired: replicas=3"]
        D2["Observe: 2 running"]
        D3["Compare: 2 < 3"]
        D4["Act: create 1 more"]
        D1 --> D2 --> D3 --> D4 --> D2
    end

Three controllers — one pattern¶

In Shepherd, every controller follows the exact same pattern:

ReplicationController¶

func (rc *ReplicationController) reconcileDeployment(dep *Deployment) {
    // Observe
    matchingPods := findPodsByLabels(dep.Spec.Selector)
    current := len(matchingPods)

    // Compare
    desired := dep.Spec.Replicas

    // Act
    if current < desired {
        // Create pods
    } else if current > desired {
        // Delete extras
    }
}

ServiceController¶

func (sc *ServiceController) reconcileService(svc *Service) {
    // Observe
    pods := findPodsByLabels(svc.Spec.Selector)

    // Compare + Act
    var endpoints []string
    for _, pod := range pods {
        if pod.Status.Phase == PodRunning && pod.Status.PodIP != "" {
            for _, port := range svc.Spec.Ports {
                endpoints = append(endpoints,
                    fmt.Sprintf("%s:%d", pod.Status.PodIP, port.TargetPort))
            }
        }
    }
    svc.Status.Endpoints = endpoints
}

NodeController¶

func (nc *NodeController) reconcile() {
    // Observe
    nodes := listAllNodes()
    for _, node := range nodes {
        // Compare
        if time.Since(node.Status.LastHeartbeat) > 30*time.Second {
            // Act
            node.Status.Condition = NodeNotReady
        }
    }
}

Why this works¶

Self-healing — no need to react to specific failures, the controller fixes the state on its own
Idempotency — run reconcile 10 times, the result is the same
Decoupling — the controller doesn't know who changed the state or why
Simplicity — each controller is very simple because it does only one thing: compare and fix

What you should know¶

Eventual consistency means delay. A pod doesn't become Running instantly. Time passes between "desired state changed" and "actual state caught up." For critical systems, this might be unacceptable.

Two more traps. A reconcile that compares states with == can get stuck flapping: the defaults the server adds to an object don't match what you sent, so the controller keeps "fixing" what's already correct. And there's no backoff: if act keeps failing (no room on the nodes), the loop spins at full speed, burning CPU and log lines.

And one more thing that rarely gets said out loud: "self-healing" heals nothing. When you delete a pod, the controller doesn't bring that pod back — it creates a new one, with a different name, a different IP, and an empty disk. The state converges at the level of counts, not at the level of a specific instance. This works exactly as long as your application treats itself as cattle rather than a pet.

💡 Fun facts¶

The term "reconciliation loop" didn't come from Kubernetes but from control theory and robotics: it's a classic control loop, like a thermostat comparing desired and actual temperature.
Kubernetes has no single "orchestrator." kube-controller-manager is dozens of independent controllers (Deployment, ReplicaSet, Node, Job...), each running its own reconcile over shared state in etcd.
The declarative model came into Kubernetes straight from Borg and Omega — Google's internal systems. Omega was the first to make shared state that all controllers read and patch optimistically, with no central dispatcher.
"Level-triggered, not edge-triggered" is a mantra of the Kubernetes designers. A controller reacts to the current state of the world, not to a change event. So a lost event doesn't break anything: the next reconcile still sees the discrepancy.
The idea predates Kubernetes by two decades. Mark Burgess's CFEngine (1993) introduced "convergent operators": operations you can apply forever, with the system only ever moving closer to the described state. Promise Theory grew out of that, and Puppet, Chef, and Terraform grew out of Promise Theory. Kubernetes isn't the inventor here — it's the most visible heir.
metadata.generation and status.observedGeneration are the built-in answer to "has the controller even seen my change?" generation only increments when spec changes, never on a status write. The controller copies it into observedGeneration after reconciling. That pair of numbers is what kubectl rollout status runs on — without it, it would show you the previous rollout's status and report success.
spec and status are physically separate API endpoints. The status subresource means kubectl apply cannot write status and a controller cannot write spec, and each gets its own RBAC rules. The "desired vs actual" split isn't a convention in the developer's head — it's enforced server-side.
Real Kubernetes controllers don't poll the API every N seconds. They keep a local cache via an informer and listen on watch. There is a periodic resync, but it's a safety net for lost events, and by default it's measured in hours (typically 10–12 with jitter), not seconds. Which means the time.Ticker loop I wrote in Shepherd is precisely what production controllers deliberately avoid.
The most common GitOps incident is two controllers fighting over one field. The HPA sets spec.replicas: 7, Argo CD sees the drift from Git and puts back 3, the HPA sets 7 again. Neither is wrong; both are doing their job. That's exactly why Argo CD ships ignoreDifferences with spec/replicas as the textbook example.
The defaults trap used to bite kubectl itself. To tell "you deleted this field" apart from "you just didn't mention it," kubectl apply stored your previous YAML as a whole JSON string in the kubectl.kubernetes.io/last-applied-configuration annotation and did a three-way merge against it. Server-Side Apply removed that crutch: the server now remembers which manager owns which field.

What I figured out while digging into this¶

The thing that retrained me most was realizing reconcile should be dumb and repeatable, not clever. At first you're tempted to write "if event X arrives, do Y." But the moment you start catching events, you're obligated not to lose them — and that means queues, retries, deduplication. When the controller simply looks each time at "how many are there, how many should there be," all that complexity disappears. A lost tick breaks nothing: the next one picks it up. It's not about elegant code, it's about being able to reason about the system in your head.

What could be improved¶

Replace the fixed interval with an event-driven loop: a Watch API instead of polling every N seconds. Reconcile fires only on a real change, not idly.
Add exponential backoff and rate limiting to the reconcile queue — like the workqueue in client-go. Failed attempts get deferred instead of spinning at full speed.
Record in Status not just current numbers but also conditions (Progressing, Available) and observedGeneration — so a client can tell "still working" from "stuck."
Make reconcile genuinely idempotent with respect to defaults: normalize the observed state before comparing, to kill the flapping.

Try it yourself¶

# Delete a pod — the controller will restore it:
sheepctl delete pod web-0
sleep 10 && sheepctl get pods
# Change replicas — the controller will adjust:
sheepctl scale deployment web --replicas 1
sleep 10 && sheepctl get pods

Next up — Label Selectors: why labels are better than foreign keys for linking resources.

Resources¶

Kubernetes controllers — desired vs actual state, with the thermostat analogy right there in the docs
Kubernetes objects: spec and status — where the desired/actual split comes from
API conventions — conditions, observedGeneration, and the rules for the status subresource
Writing Controllers — the official SIG API Machinery guide, including "don't assume the event arrived"
Kubernetes architecture (design proposals archive) — the primary source for "level-triggered, not edge-triggered"
Declarative application management in Kubernetes — Brian Grant's essay on why declarative beats scripts
Efficient detection of changes — watch, resourceVersion, bookmark events
client-go informers — a local cache instead of polling, and what the resync period is actually for
client-go workqueue — the rate limiting and exponential backoff my loop doesn't have
controller-runtime — all of the above, packaged into one Reconciler
Kubebuilder: controller implementation — a real reconcile, written step by step
Server-Side Apply — field managers instead of last-applied-configuration and three-way merges
Operator pattern — the same loop, moved into your own domain
Argo CD: diffing — ignoreDifferences, and how HPA and GitOps are kept from fighting over replicas
Borg, Omega and Kubernetes — the history of declarative orchestration from the people who built it
Large-scale cluster management at Google with Borg — where it all started
Omega: flexible, scalable schedulers — shared state and optimistic concurrency with no central dispatcher
Promise Theory — Mark Burgess's formalization, the root of all convergent configuration
CFEngine — 1993, convergent operators, the ancestor of desired state
Control loop — the same thermostat, in control theory terms
Patterns of Distributed Systems — relevant patterns

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Push to Your Own Registry | Next: Label Selectors