Skip to content

Replication Controller: Scale Up and Scale Down

Replication Controller: Scale Up and Scale Down

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

In the previous part, we saw the reconciliation loop. Now let's look at how the ReplicationController creates pods for a Deployment and manages their count.

Creating a Pod from a Deployment

func (rc *ReplicationController) createPodForDeployment(
    dep *Deployment, index int) {
    pod := &Pod{
        Kind: "Pod",
        Metadata: ObjectMeta{
            Name: fmt.Sprintf("%s-%d",
                dep.Metadata.Name, index),
            Namespace: dep.Metadata.Namespace,
            UID:       generateUID(),
            Labels:    mergeLabels(
                dep.Spec.Template.Metadata.Labels,
                dep.Spec.Selector),
            CreatedAt: time.Now(),
        },
        Spec: dep.Spec.Template.Spec,
        Status: PodStatus{Phase: PodPending},
    }

    rc.store.CreatePod(pod)

    rc.store.RecordEvent(Event{
        Type:    "Normal",
        Reason:  "Created",
        Message: fmt.Sprintf("Created pod %s for deployment %s",
            pod.Metadata.Name, dep.Metadata.Name),
        Object: "deployment/" + dep.Metadata.Name,
    })

    // Scheduling
    go rc.scheduler.SchedulePod(pod)
}

Notice the naming: web-0, web-1, web-2. Like a StatefulSet in Kubernetes, just simpler.

The pod's labels are a merge of the deployment's template and selector. This is what ties pods to the deployment.

graph TB
    DEP["Deployment: web<br/>selector: app=web<br/>replicas: 3"]

    DEP --> P0["Pod: web-0<br/>labels: app=web"]
    DEP --> P1["Pod: web-1<br/>labels: app=web"]
    DEP --> P2["Pod: web-2<br/>labels: app=web"]

A Deployment doesn't store a list of its pods. It finds them through label selectors: "all pods with label app=web in my namespace are mine."

func mergeLabels(sets ...map[string]string) map[string]string {
    result := make(map[string]string)
    for _, s := range sets {
        for k, v := range s { result[k] = v }
    }
    return result
}

Scale Up

If there are fewer pods than spec.replicas:

if current < desired {
    for i := 0; i < desired-current; i++ {
        rc.createPodForDeployment(dep, current+i)
    }
    rc.logger.Printf(
        "replication controller: scaled up %s from %d to %d",
        dep.Metadata.Name, current, desired)
}

New pods are created with index current+i. The Scheduler assigns them to nodes automatically.

Scale Down

If there are more pods than needed:

if current > desired {
    for i := 0; i < current-desired; i++ {
        pod := matchingPods[len(matchingPods)-1-i]
        rc.store.DeletePod(
            pod.Metadata.Namespace, pod.Metadata.Name)
    }
}

We delete from the end of the list. In real Kubernetes, there's more sophisticated logic: Failed pods are deleted first, then Pending, then those with the shortest uptime.

Updating Deployment Status

ready := 0
for _, pod := range matchingPods {
    if pod.Status.Phase == PodRunning { ready++ }
}
dep.Status.Replicas = len(matchingPods)
dep.Status.ReadyReplicas = ready
dep.Status.AvailableReplicas = ready
rc.store.UpdateDeployment(dep)

ReadyReplicas is how many pods are actually Running. sheepctl get deployments will show something like 3/3 Ready.

The Full Flow from sheepctl to Container

sequenceDiagram
    participant U as sheepctl
    participant API as API Server
    participant RC as ReplicationController
    participant S as Scheduler
    participant A as Agent
    participant R as Sheep Runtime

    U->>API: POST /api/v1/deployments<br/>{replicas: 3}
    API->>API: Save to Store
    RC->>RC: reconcile() every 5s
    RC->>API: Sees: 0 pods, needs 3
    RC->>API: CreatePod(web-0)
    RC->>API: CreatePod(web-1)
    RC->>API: CreatePod(web-2)
    S->>S: reconcile() every 2s
    S->>API: Pod web-0 → node-1
    A->>A: reconcile() every 3s
    A->>R: Create + Start container
    A->>API: Pod web-0: Running

Everything is asynchronous. Each component works at its own pace and reads state from the Store.

What We Skipped

Scale down removes pods from the end of the list without considering their state. In Kubernetes, a ReplicaSet first deletes Failed pods, then Pending, then those with the shortest uptime. Our approach is simpler but less smart.

  • Index-based naming web-0..web-N is dangerous: delete web-1, and on scale-up the controller counts current and recreates a pod with the same name -- or steps on a stale record. That's why a Kubernetes ReplicaSet gives pods random suffixes (web-x7k2p) instead of sequential indices.
  • If two selectors in different deployments overlap, a single pod can "belong" to both -- and two controllers will start alternately creating and deleting it. Owner references in Kubernetes exist precisely to guard against this.

💡 Fun facts

  • Sequential names like web-0, web-1 are actually the StatefulSet model, not Deployment. A regular Kubernetes Deployment creates a ReplicaSet with a random hash in the pod names precisely because stable identity is only needed by stateful workloads (databases, queues), while stateless replicas are interchangeable.
  • A Deployment in Kubernetes doesn't manage pods directly -- there's an intermediate object, the ReplicaSet, between them. That layer is exactly what makes rolling updates work: a new ReplicaSet is created and the old one is scaled down to zero. Our controller skips this level.
  • The "deployment → pods" link in Kubernetes rests not only on the label selector but also on the ownerReferences field in each pod. This is what gives you cascading deletion (garbage collection): delete the deployment and its pods get cleaned up automatically.
  • The scale-down deletion order in Kubernetes is defined by an explicit cost function: Pending/unscheduled first, then not-Ready, then younger pods, then those on nodes with more replicas -- to thin out the cluster evenly.
  • ReplicationController was literally the first controller Kubernetes ever shipped -- our class name is a direct homage. It was later superseded by ReplicaSet (which gained set-based selectors), but kubectl still answers to the rc shortname for backward compatibility.
  • kubectl scale doesn't trigger any special "scale" operation -- it just patches the spec.replicas field via a dedicated /scale subresource. The controller picks up the new number on its next reconcile, exactly like ours does. The same field is what a HorizontalPodAutoscaler rewrites automatically based on CPU/memory metrics -- the controller has no idea it's being autoscaled.
  • replicas: 0 is perfectly valid and is the standard way to "pause" a workload without deleting its config. Scale-to-zero is the whole foundation of serverless platforms like Knative and KEDA, which spin a Deployment down to nothing when idle and back up on the first request.

What I figured out while digging into this

The least obvious thing turned out to be that a deployment doesn't really "own" its pods in any direct sense -- it doesn't hold them in a list, it re-finds them every time through the label selector. At first that felt fragile: what if someone slaps the same label on by hand? But then I realized that this very loose coupling is what makes the system resilient -- the controller doesn't care where a pod came from, only that the labels match. The downside is that selectors are easy to overlap by accident, and then two controllers end up fighting over the same pods.

What could be improved

  • Replace sequential indices with random name suffixes so that scale-up after a mid-list deletion doesn't conflict.
  • Implement a smart scale-down order: Failed first, then Pending, then the youngest pods.
  • Add an intermediate ReplicaSet layer to get a foundation for rolling updates and rollbacks.
  • Introduce ownerReferences and cascading deletion instead of relying on the selector alone.

Try It Yourself

# Scale deployment:
sheepctl scale deployment web --replicas 5
sheepctl get pods     # you'll see 5 pods
sheepctl scale deployment web --replicas 2
sheepctl get pods     # you'll see 2 pods

Scaling works. Next up -- Service Discovery: how a Service automatically finds its pods.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Reconciliation Loop | Next: Service Discovery