Skip to content

Node Agent: kubelet

Node Agent: kubelet

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

The Agent is Shepherd's kubelet. It runs on every node, registers with the API Server, sends heartbeats, and starts containers for assigned pods.

Structure

type Agent struct {
    nodeName  string
    apiAddr   string
    mgr       *container.Manager
    logger    *log.Logger
    capacity  NodeResources
}

The Agent knows its name, the API Server address, and has access to the container.Manager (Sheep runtime).

Registration with Retry

func (a *Agent) Run(stopCh <-chan struct{}) error {
    a.mgr.Init()

    // Retry for standalone mode
    for attempt := 0; attempt < 10; attempt++ {
        if err := a.register(); err == nil {
            break
        } else if attempt == 9 {
            return fmt.Errorf("register after 10 attempts: %w", err)
        }
        time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)
    }

    heartbeat := time.NewTicker(10 * time.Second)
    reconcile := time.NewTicker(3 * time.Second)

    for {
        select {
        case <-stopCh: return nil
        case <-heartbeat.C: a.sendHeartbeat()
        case <-reconcile.C: a.reconcilePods()
        }
    }
}

Retry is needed for standalone mode, where the API Server and Agent start at the same time. The API Server may not be ready yet.

Node Registration

func (a *Agent) register() error {
    hostname, _ := os.Hostname()
    node := Node{
        Kind: "Node",
        Metadata: ObjectMeta{
            Name: a.nodeName,
            Labels: map[string]string{
                "hostname": hostname,
                "os":       runtime.GOOS,
                "arch":     runtime.GOARCH,
            },
        },
        Spec: NodeSpec{Address: a.nodeName},
        Status: NodeStatus{
            Condition:   NodeReady,
            Capacity:    a.capacity,
            Allocatable: a.capacity,
        },
    }
    return a.post("/api/v1/nodes", node)
}

Labels automatically include hostname, OS, and architecture. A pod's nodeSelector can use these labels.

Resource Detection

func detectCapacity() NodeResources {
    var memTotal int64
    data, err := os.ReadFile("/proc/meminfo")
    if err == nil {
        fmt.Sscanf(string(data), "MemTotal: %d kB", &memTotal)
        memTotal *= 1024
    }
    if memTotal == 0 {
        memTotal = 2 * 1024 * 1024 * 1024 // 2GB default
    }

    return NodeResources{
        CPU:    int64(runtime.NumCPU()) * 1000, // millicores
        Memory: memTotal,
        Pods:   110,
    }
}

Reads memory from /proc/meminfo, CPU from runtime.NumCPU(). 110 pods per node is the default Kubernetes limit.

Pod Reconciliation

func (a *Agent) reconcilePods() {
    pods, _ := a.listMyPods()

    for _, pod := range pods {
        if pod.Spec.NodeName != a.nodeName { continue }

        switch pod.Status.Phase {
        case PodPending:
            a.startPod(pod)
        case PodRunning:
            a.checkPod(pod)
        }
    }
}

Every 3 seconds, the Agent checks its pods. Pending ones get started. Running ones get health-checked.

Starting a Pod

func (a *Agent) startPod(pod *Pod) {
    var statuses []ContainerStatus

    for _, cs := range pod.Spec.Containers {
        var env []string
        for k, v := range cs.Env {
            env = append(env, fmt.Sprintf("%s=%s", k, v))
        }

        opts := container.RunOpts{
            Name:  fmt.Sprintf("%s-%s",
                pod.Metadata.Name, cs.Name),
            Image:   cs.Image,
            Command: cs.Command,
            Config: container.Config{
                Hostname: pod.Metadata.Name,
                Env: env,
                Memory: cs.Resources.Memory,
            },
        }

        c, err := a.mgr.Create(opts)
        if err != nil {
            statuses = append(statuses, ContainerStatus{
                Name: cs.Name, State: "failed"})
            continue
        }

        a.mgr.Start(c.ID)

        statuses = append(statuses, ContainerStatus{
            Name: cs.Name, ContainerID: c.ID,
            Ready: true, State: "running"})
    }

    // Update IP
    if pod.Status.PodIP == "" {
        for _, cs := range statuses {
            if cs.ContainerID != "" {
                c, _ := a.mgr.Get(cs.ContainerID)
                if c != nil && c.Network != nil {
                    pod.Status.PodIP = c.Network.IPAddress
                    break
                }
            }
        }
        if pod.Status.PodIP == "" {
            pod.Status.PodIP = "127.0.0.1"
        }
    }

    allReady := true
    for _, s := range statuses {
        if !s.Ready { allReady = false; break }
    }

    if allReady {
        pod.Status.Phase = PodRunning
    } else {
        pod.Status.Phase = PodFailed
    }

    a.updatePodStatus(pod)
}

graph TD
    A["Pod Pending<br/>NodeName = this node"] --> B["For each container in spec"]
    B --> C["Create container"]
    C --> D["Start container"]
    D --> E["Get IP from network"]
    E --> F{"All Ready?"}
    F -->|Yes| G["Pod = Running"]
    F -->|No| H["Pod = Failed"]

The Agent takes the IP address of the first container and sets it as PodIP. This is a simplification -- in Kubernetes, all containers in a pod share one network namespace.

Limitations

The Agent creates a separate container for each container spec in the pod. In Kubernetes, all containers in a pod share one network namespace (via the pause container). In Shepherd, each container gets its own IP, which breaks the pod networking model.

A few more sharp edges:

  • detectCapacity() reads /proc/meminfo -- that's Linux-specific. On macOS the file doesn't exist, so the code silently falls back to the 2 GB default. No scheduler will ever learn that the number is made up.
  • The Agent only reconciles Pending and Running. If a pod is deleted from the API while the Agent wasn't looking, the container keeps running -- an orphan. There's no reconciliation of "what's actually running" against "what should be running."

💡 Fun facts

  • The name kubelet is "kubernetes" + "-let" (like "applet", "servlet"): a small agent. And despite the name, it's one of the largest and most complex components of Kubernetes -- far from 350 lines.
  • The real kubelet hasn't run containers itself for a long time. It talks over CRI (Container Runtime Interface) -- a gRPC protocol -- to a runtime like containerd or CRI-O. Our Agent, calling mgr.Create() directly, is closer to how kubelet worked before ~1.5, when Docker was wired in internally.
  • The pause container in Kubernetes (also "infra container") is a tiny program that does nothing but sleep and hold the network namespace. Its sole job is to outlive restarts of the application containers so the pod IP doesn't change. The pause image is one of the most frequently run containers in the world -- roughly 700 KB on disk and about 250 KB of RAM.
  • 110 pods per node isn't a physical limit -- it's a conservative kubelet default (--max-pods). It's tied more to pressure on the control plane and network than to the node's own resources. On large clusters, the scheduler and API Server feel every new pod object long before the node runs out of CPU.
  • The real kubelet's main loop is syncLoop: it watches the API for pod changes and periodically re-syncs everything anyway. That combination of watch + reconcile is why missed events rarely matter -- the same insight as our 3-second ticker, just at production scale.
  • Inside syncLoop, the Pod Lifecycle Event Generator (PLEG) watches container runtime events and maps them back to pods. Without PLEG, kubelet wouldn't know a container died until the next full sync -- a dedicated bridge between "runtime said something" and "pod status should change."
  • Kubernetes also supports static pods: manifest files dropped into /etc/kubernetes/manifests/ that the kubelet runs without the scheduler. The control plane sees them as mirror pods. Shepherd has no equivalent -- every pod must come through the API.
  • The kubelet concept traces back to Borg at Google: the borglet on each machine was the ancestor of today's node agent. Kubernetes kept the pattern -- one daemon per node that makes local reality match cluster intent.
  • When a node joins a real cluster, the kubelet goes through TLS bootstrap: it gets a client certificate signed by the control plane so only authorized agents can register and update node state. Shepherd skips that entirely -- any process that knows the API address can pretend to be a node.

What I figured out while digging into this

The biggest insight: the Agent doesn't "manage" pods, it reconciles them. There's no "start this pod" command -- just a loop that every three seconds asks "which pods are assigned to me?" and brings reality up to the desired state. At first the polling-instead-of-push bugged me. But that's exactly what makes the Agent self-healing: miss an event, and the next tick fixes everything anyway. Push without reconcile would be more fragile.

I also understood why the retry on registration is needed. The first standalone run kept failing because the Agent knocked on an API Server that hadn't come up yet. It was a reminder that in distributed systems "the other component isn't ready yet" is the norm, not an error.

What could be improved

  • Add a pause container per pod and share one network namespace across containers -- then PodIP becomes real instead of the first container's IP.
  • Move to a CRI-like interface instead of direct container.Manager calls -- then the runtime could be swapped out.
  • Implement real state reconciliation: find orphan containers (running, but with no pod in the API) and clean them up.
  • Cross-platform detectCapacity(): a separate branch for macOS (sysctl hw.memsize) instead of silently falling back to 2 GB.

Try It Yourself

# In multi-node mode:
# Terminal 1 (control plane):
shepherd --mode server --addr :9876
# Terminal 2 (worker):
shepherd --mode agent --node-name worker-1 --api-addr localhost:9876
sheepctl nodes  # you'll see worker-1

The Agent works. Next up -- the full journey of a pod from creation to Running.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Node Health | Next: Pod Lifecycle