Skip to content

Async Scheduling: Why a Pod Is Created as Pending

Async Scheduling: Why a Pod Is Created as Pending

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

When you run sheepctl apply pod.json, the API Server returns 201 Created immediately. The pod isn't running yet, not even scheduled. It's Pending. This isn't a bug - it's a deliberate decision that makes the system resilient to failures.

What happens in the API Server

func (api *APIServer) createPod(w http.ResponseWriter,
    r *http.Request) {
    var pod Pod
    json.NewDecoder(r.Body).Decode(&pod)

    pod.Kind = "Pod"
    if pod.Metadata.Namespace == "" {
        pod.Metadata.Namespace = "default"
    }
    if pod.Metadata.UID == "" {
        pod.Metadata.UID = generateUID()
    }
    pod.Metadata.CreatedAt = time.Now()
    pod.Status.Phase = PodPending  // Always Pending!

    api.store.CreatePod(&pod)

    api.store.RecordEvent(Event{
        Type:    "Normal",
        Reason:  "Created",
        Message: fmt.Sprintf("Pod %s created", pod.Metadata.Name),
        Object:  "pod/" + pod.Metadata.Name,
    })

    go api.scheduler.SchedulePod(&pod) // fire-and-forget

    respondJSON(w, http.StatusCreated, pod)
}

Three key points:
1. Phase = PodPending - always, no exceptions
2. go api.scheduler.SchedulePod() - asynchronous call
3. respondJSON(201) - right after saving, without waiting for scheduling

Synchronous vs asynchronous approach

graph TB
    subgraph "Synchronous approach (not what we do)"
        S1["POST /pods"] --> S2["Save pod"]
        S2 --> S3["Find node"]
        S3 --> S4["Wait for agent"]
        S4 --> S5["Wait for container start"]
        S5 --> S6["Response: 201 Running"]
        S3 -->|"No nodes"| S7["Timeout 30s"]
        S7 --> S8["Response: 503"]
    end

    subgraph "Asynchronous approach (what we do)"
        A1["POST /pods"] --> A2["Save pod (Pending)"]
        A2 --> A3["Response: 201 Pending"]
        A2 -.->|"async"| A4["Scheduler picks up"]
        A4 -.-> A5["Agent starts containers"]
    end

The synchronous approach means the HTTP request blocks for seconds or even minutes. If there are no nodes - timeout. If the agent is slow - the client waits. If the API Server is under load - all workers are busy waiting.

Asynchronous: the API Server responds in milliseconds. Scheduling, starting containers - all happens in the background.

Three stages of Pending

A pod in the Pending state can be in different situations:

stateDiagram-v2
    [*] --> PendingNoNode : API created it
    PendingNoNode --> PendingWithNode : Scheduler assigned it
    PendingWithNode --> Running : Agent started it
    PendingNoNode --> PendingNoNode : Scheduler couldn't find a node

    note right of PendingNoNode
        NodeName = ""
        Waiting for scheduler
    end note

    note right of PendingWithNode
        NodeName = "node-1"
        Waiting for agent
    end note

How do you tell the difference? The pod.Spec.NodeName field:
- Empty - scheduler hasn't looked yet or couldn't find a node
- Filled - scheduler assigned it, agent hasn't started it yet 

// Scheduler only looks for pods without a node
func (s *Scheduler) reconcile() {
    pods, _ := s.store.ListPods("")
    for _, pod := range pods {
        if pod.Status.Phase == PodPending &&
            pod.Spec.NodeName == "" {
            s.SchedulePod(pod)
        }
    }
}

Resilience through reconciliation

Here's why go scheduler.SchedulePod() in createPod is fire-and-forget. Even if that goroutine crashes:

// Scheduler reconciliation loop - every 2 seconds
func (s *Scheduler) Run(stopCh <-chan struct{}) {
    ticker := time.NewTicker(2 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-stopCh: return
        case <-ticker.C:
            s.reconcile() // picks up any Pending pods
        }
    }
}

The reconciliation loop will find the Pending pod without a NodeName and schedule it. No retry logic needed, no error handling for the goroutine - the loop fixes the state within 2 seconds.

Timeline of pod creation

T+0ms:    sheepctl apply → POST /api/v1/pods
T+1ms:    API Server saves pod (Phase: Pending, NodeName: "")
T+2ms:    API Server responds 201 Created
T+2ms:    go scheduler.SchedulePod() fired
T+100ms:  Scheduler assigns pod to node-1
T+3000ms: Agent reconcile tick - sees Pending pod on its node
T+3500ms: Agent creates and starts container
T+3600ms: Agent updates pod (Phase: Running, PodIP: 10.20.0.2)

From creation to Running - about 3.5 seconds. For a production pipeline, that's acceptable. For real-time - no, but container orchestration doesn't need real-time.

There's one thing

Eventual consistency means that right after sheepctl apply, the client sees Pending. If a script does apply and immediately checks the state - it'll see Pending and might conclude something is broken. You need polling or watch:

# Bad approach:
sheepctl apply -f pod.json
sheepctl get pods  # still Pending!

# Right approach:
sheepctl apply -f pod.json
sleep 5 && sheepctl get pods  # Running

In Kubernetes, kubectl wait --for=condition=Ready pod/web-0 solves this through the Watch API. In Shepherd there's no Watch API, so it's polling only.

A couple more traps. go scheduler.SchedulePod() plus the reconcile loop means the same pod can get picked up by both paths almost simultaneously - without a NodeName == "" check inside a critical section, it can be scheduled twice. And Pending isn't forever in the user's head, but it is in the system: a pod that lacks the resources it needs can hang in Pending indefinitely, silently, until someone looks at events.

💡 Fun facts

  • In Kubernetes the scheduler and the kubelet are separate components that don't even talk directly. The scheduler only sets pod.spec.nodeName via the API Server, and the kubelet on the node notices "oh, this pod is mine" and starts it. Exactly like our NodeName.
  • And the scheduler doesn't even PATCH the pod. Assignment goes through a dedicated subresource - POST /api/v1/namespaces/{ns}/pods/{name}/binding. A Binding object is effectively an assignment letter that the API Server unpacks into a spec.nodeName write. A separate subresource means separate RBAC: you can let a component place pods without granting it the right to rewrite their whole spec.
  • This split into "assign" and "run" is an example of the optimistic concurrency from the Omega paper: components don't block each other, they act independently and reconcile the state afterward.
  • The optimism is literal: kube-scheduler has an assume step - it writes the pod into its local cache as already placed before the bind call even returns. Otherwise, in the time one HTTP request is in flight, it would happily sell the same node memory to the next ten pods. If the bind fails, the pod is forgetten and goes back into the queue.
  • Pending in Kubernetes isn't a single cause but a whole family: no resources, failed affinity, a taint without a toleration, an unmounted volume. From the outside it's one phase, inside it's dozens of scenarios. The real reason lives not in phase but in status.conditions - a PodScheduled=False condition with reason Unschedulable.
  • kube-scheduler deliberately doesn't score every node. The percentageOfNodesToScore knob makes it stop once it has found "enough" feasible ones (adaptive, but never below 5% on large clusters). That's placement accuracy traded directly for latency - on a 5000-node cluster, walking all of them is simply too expensive.
  • SIG Scalability puts a number on this: 99% of pods without stateful dependencies should start in under 5 seconds - with image pull time deliberately excluded from the SLO, because it isn't the orchestrator's to control.
  • 201 Created right after the write, without waiting for the launch, is the same principle as HTTP 202 Accepted: "got it, I'll process it later." Async APIs predate Kubernetes by decades.

What I figured out while digging into this

The biggest insight: fire-and-forget looks sloppy, but it's actually more honest than a synchronous call. A synchronous apply that waits for Running lies to the client about reliability - let the agent crash mid-launch and you get a 200 for a pod that doesn't exist. Async with a reconcile loop promises nothing up front: it just guarantees the state eventually converges. I stopped thinking of that goroutine as "launching the pod" and started seeing it as a hint to the scheduler - a nice latency optimization that everything works fine without, thanks to the loop.

What could be improved

  • Eliminate the race: do scheduling only through the reconcile loop, and keep go SchedulePod() as an optional fast-path with a check-and-claim (a CAS on NodeName).
  • Add a reason to Status: why the pod is Pending (Unschedulable, InsufficientResources) plus an event, so you don't have to guess.
  • Implement sheepctl wait --for=Running on top of polling with a timeout - even without a Watch API, this closes 90% of the pain in scripts.
  • Add backoff to scheduling: if no node is found, don't hammer reconcile every two seconds forever, grow the interval instead.

Try it yourself

# Create a pod and immediately check its state:
sheepctl apply -f examples/pod.json
sheepctl get pods  # Phase: Pending (not yet scheduled)
sleep 2
sheepctl get pods  # Phase: Pending (scheduled, agent hasn't started yet)
sleep 3
sheepctl get pods  # Phase: Running
sheepctl events    # you'll see Created → Scheduled → Running

Next up - Graceful Shutdown: how a single close(channel) stops all goroutines.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Label Selectors | Next: Graceful Shutdown