Node Health: Heartbeat and Failure Detection

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

---

How does the control plane know a node is alive? Through heartbeats. The agent on each node sends a heartbeat every 10 seconds. If no heartbeat arrives for 30 seconds, the NodeController marks the node as NotReady. The Scheduler stops assigning pods to it.

Two Sides of the Mechanism

sequenceDiagram
    participant A as Agent (node-1)
    participant API as API Server
    participant NC as NodeController

    loop Every 10 seconds
        A->>API: PUT /api/v1/nodes/node-1<br/>{condition: Ready, lastHeartbeat: now, podCount: 5}
        API->>API: Save to BoltDB
    end

    Note over A: Agent crashes (crash, network, etc.)

    loop Every 10 seconds
        NC->>API: GET /api/v1/nodes
        NC->>NC: time.Since(lastHeartbeat) > 30s?
        NC->>API: PUT node-1 {condition: NotReady}
        NC->>API: RecordEvent("Warning", "NodeNotReady")
    end

Agent -- The Heartbeat Sender

func (a *Agent) Run(stopCh <-chan struct{}) error {
    // ... registration ...

    heartbeat := time.NewTicker(10 * time.Second)
    defer heartbeat.Stop()

    reconcile := time.NewTicker(3 * time.Second)
    defer reconcile.Stop()

    for {
        select {
        case <-stopCh:
            return nil
        case <-heartbeat.C:
            a.sendHeartbeat()
        case <-reconcile.C:
            a.reconcilePods()
        }
    }
}

func (a *Agent) sendHeartbeat() {
    node, err := a.getNode()
    if err != nil {
        a.logger.Printf("agent: get node error: %v", err)
        return
    }

    // Update state
    containers := a.mgr.List(false) // running only
    node.Status.PodCount = len(containers)
    node.Status.LastHeartbeat = time.Now()
    node.Status.Condition = NodeReady

    if err := a.put("/api/v1/nodes/"+a.nodeName, node); err != nil {
        a.logger.Printf("agent: heartbeat error: %v", err)
    }
}

The heartbeat updates three things:
1. PodCount -- how many containers are actually running (the scheduler uses this for scoring)
2. LastHeartbeat -- timestamp (the NodeController checks freshness)
3. Condition = Ready -- each heartbeat confirms "I'm alive"

NodeController -- The Failure Detector

type NodeController struct {
    store  *Store
    logger *log.Logger
}

func (nc *NodeController) Run(stopCh <-chan struct{}) {
    ticker := time.NewTicker(10 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-stopCh: return
        case <-ticker.C: nc.reconcile()
        }
    }
}

func (nc *NodeController) reconcile() {
    nodes, err := nc.store.ListNodes()
    if err != nil { return }

    for _, node := range nodes {
        if time.Since(node.Status.LastHeartbeat) >
            30*time.Second {
            if node.Status.Condition != NodeNotReady {
                nc.logger.Printf(
                    "node controller: node %s is not ready "+
                    "(last heartbeat: %s ago)",
                    node.Metadata.Name,
                    time.Since(node.Status.LastHeartbeat).
                        Round(time.Second))

                node.Status.Condition = NodeNotReady
                nc.store.UpdateNode(node)

                nc.store.RecordEvent(Event{
                    Type:    "Warning",
                    Reason:  "NodeNotReady",
                    Message: fmt.Sprintf(
                        "Node %s heartbeat timeout",
                        node.Metadata.Name),
                    Object:    "node/" + node.Metadata.Name,
                    Timestamp: time.Now(),
                })
            }
        }
    }
}

The if node.Status.Condition != NodeNotReady check prevents event spam. Without it, the NodeController would write a new Warning event every 10 seconds for the same dead node.

Node State Machine

stateDiagram-v2
    [*] --> Ready : Agent registers
    Ready --> Ready : Heartbeat (< 30s)
    Ready --> NotReady : Heartbeat timeout (> 30s)
    NotReady --> Ready : Heartbeat received
    NotReady --> [*] : Node deleted

    note right of Ready
        Scheduler assigns new pods
        Agent is working normally
    end note

    note right of NotReady
        Scheduler ignores the node
        Existing pods are not moved
    end note

Impact on the Scheduler

The Scheduler checks both condition and timestamp:

func (s *Scheduler) filterNodes(nodes []*Node,
    pod *Pod) []*Node {
    var feasible []*Node
    for _, node := range nodes {
        if node.Status.Condition != NodeReady {
            continue // skip NotReady
        }
        if time.Since(node.Status.LastHeartbeat) >
            30*time.Second {
            continue // stale heartbeat
        }
        // ... other checks
        feasible = append(feasible, node)
    }
    return feasible
}

The double check is needed because the NodeController has its own interval (10s). There can be a situation where the heartbeat is already stale, but the NodeController hasn't marked the node NotReady yet.

Timings

What Happens	Time
Agent sends heartbeat	every 10s
NodeController checks	every 10s
Heartbeat timeout	30s
Worst case: agent crashes -> NotReady	up to 40s (30s timeout + 10s NC interval)
Best case	~30s

Gotchas

30 seconds is a compromise. Less means more false positives: a 15-second network glitch and the node "went down." More means slower reaction to a real failure.

In Kubernetes, the default is 40 seconds. And even after NotReady, pods aren't deleted immediately -- there's a 5-minute grace period. In Shepherd, we don't move pods off NotReady nodes at all. If the node comes back -- the pods come back with it. If not -- the ReplicationController creates new ones on other nodes (because it sees fewer pods than desired).

One more thing: if the network between the Agent and API Server drops, the Agent keeps running and serving containers. Only the heartbeat doesn't get through. The control plane considers the node dead, even though containers are working fine. This is a split-brain situation, and there's no simple fix.

A couple more things worth watching:

• time.Since(lastHeartbeat) is computed against the control plane's clock, while the timestamp is set by the Agent. If the clocks drift apart (clock skew), a node can "die" or "come back" for no real reason. In a real kubelet, this is exactly why Lease objects are updated separately from status.
• The heartbeat carries the whole node object via PUT. Without optimistic locking (resourceVersion), two concurrent writes can clobber each other -- last write wins. With one Agent per node that's fine, but you can't scale it this way.

💡 Fun facts

• Kubernetes used to keep lastHeartbeatTime right in the node status -- and every kubelet rewrote the entire node object into etcd every few seconds. On large clusters this created a crushing write load. The fix was a dedicated Lease resource in the kube-node-lease namespace: a tiny object that's cheap to update. Today detection rides on the cheap Lease, while the heavy full-status update fires much less often -- the two heartbeats were deliberately split by frequency.
• The default node-monitor-grace-period in Kubernetes is around 40 seconds, but that's not "time until the node dies." It's only the threshold for flipping to NotReady. Pod eviction is driven by separate NoExecute taints with their own tolerationSeconds (300 seconds by default).
• When the node controller flips a node to NotReady, it stamps a node.kubernetes.io/not-ready:NoExecute taint, and eviction is rate-limited (--node-eviction-rate, default 0.1/s = one node every 10s). If too large a fraction of a zone goes unhealthy at once, the controller deliberately stops evicting -- the assumption being "it's probably the network, not the nodes." A built-in brake against partition-triggered eviction storms.
• A heartbeat is a classic failure detector. Distributed systems theory proves that no perfect failure detector exists in an asynchronous network (the FLP impossibility): you cannot tell "node is dead" apart from "node is slow." Any timeout is a bet.
• Smarter systems don't hardcode a single timeout. The Φ (Phi) Accrual failure detector (used by Cassandra, Akka, Hazelcast) replaces the binary dead/alive answer with a continuously rising suspicion level φ that adapts to the observed distribution of heartbeat inter-arrival times -- the threshold effectively self-tunes to the network instead of guessing 30s once.
• Not everyone centralizes detection like our NodeController polling every node. SWIM-style gossip (Serf, Consul, Hazelcast) has each node randomly probe a few peers and gossip suspicion around, so the per-node cost stays roughly constant -- which is how membership scales to thousands of nodes without a single component watching them all.
• Real Kubernetes runs on etcd, which has its own Raft heartbeats (typically every ~100 ms) between leader and followers -- a second, lower layer of "are you alive?" underneath the node heartbeats. Shepherd skips all of that: our store is a single embedded BoltDB file with no consensus and no leader, so that layer simply doesn't exist for us -- one less moving part, but also no built-in replication.

What I figured out while digging into this

While writing this part, it finally hit me that a heartbeat isn't a "liveness check" -- it's an agreement about a timeout. A heartbeat on its own guarantees nothing: the absence of a signal can mean a dead node or just network lag. We're not detecting death -- we're deciding after how much silence to call a node dead.

The second thing that clicked: the split of roles. The Agent only writes "I'm alive," while the "you're dead" decision is made by a separate component the Agent never even sees. At first I wanted the Agent to notice its own problems -- but a dead process can't report that it's dead. Someone outside has to be the judge.

What could be improved

• Move the heartbeat into a separate lightweight object (like Kubernetes' Lease) so we don't rewrite the whole node object every time -- less load on BoltDB and fewer write conflicts.
• Make the timeout a configurable flag instead of a hardcoded 30s, and add a degraded state between Ready and NotReady (one or two missed heartbeats isn't death yet).
• Add node taints and controlled eviction with a grace period instead of the "just don't touch pods on a NotReady node" approach.
• Account for clock skew: compute heartbeat "freshness" from the server's receive time, not from the timestamp the Agent stamped on it.

Try It Yourself

# Check nodes:
sheepctl nodes
# You'll see: node-1  Ready  pods: 5  last heartbeat: 3s ago

# Stop the agent (Ctrl+C or kill) and wait:
sleep 35
sheepctl nodes
# You'll see: node-1  NotReady  last heartbeat: 35s ago

sheepctl events | head -3
# Warning  NodeNotReady  node/node-1  Node node-1 heartbeat timeout

# Restart the agent - node will become Ready:
shepherd --mode agent --node-name node-1 --api-addr localhost:9876
sheepctl nodes  # Ready again

---

Health monitoring works. Next up -- Node Agent: kubelet in 350 lines.

Resources

• Nodes — node lifecycle, conditions, heartbeats
• Heartbeats and Node Leases — the Lease mechanism that replaced full status writes
• Controllers — including node controller
• Taint-based Evictions — how NotReady turns into pod eviction via tolerationSeconds
• kube-controller-manager flags — --node-monitor-grace-period, --node-eviction-rate, zone thresholds
• The Φ Accrual Failure Detector (Hayashibara et al.) — adaptive suspicion instead of a fixed timeout
• SWIM: Scalable Weakly-consistent Membership — gossip-based failure detection that scales to large clusters

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Service Discovery | Next: Node Agent