Skip to content

Goroutines for Control Loops

Goroutines for Control Loops

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

Shepherd runs 6 parallel goroutines, each on its own clock. The Scheduler looks for Pending pods every 2 seconds. The ReplicationController checks replica counts every 5. The Agent sends a heartbeat every 10 seconds and checks its pods every 3. They never talk to each other. All they share is the BoltDB Store and one stop channel.

Earlier parts covered what each controller does, and the reconciliation loop as an idea. This one is about how it actually runs in Go: tickers, select, shared state, and what happens when something breaks. The previous part was about syscalls, the bottom of the runtime. This one is about the top, the control plane.

The Pattern: ticker + select + stopCh

Every controller follows the same template:

func (s *Scheduler) Run(stopCh <-chan struct{}) {
    ticker := time.NewTicker(2 * time.Second)
    defer ticker.Stop()

    s.logger.Println("scheduler started")

    for {
        select {
        case <-stopCh:
            s.logger.Println("scheduler stopped")
            return
        case <-ticker.C:
            s.reconcile()
        }
    }
}

select blocks until one of the channels is ready. The ticker fires, we reconcile. stopCh closes, we exit. The two branches never run at the same time.

defer ticker.Stop() is mostly good hygiene now. Before Go 1.23, a ticker without Stop() kept running after the goroutine exited, a classic leak. Since Go 1.23 the garbage collector reclaims an unreferenced ticker even if nobody stopped it. Code that still has to build on older versions keeps the Stop().

Why close the channel instead of sending a value? Every receiver sees a close(stopCh), however many there are. A sent value goes to exactly one receiver. One close() stops all four controllers at once.

Starting All Controllers

func runServer(addr, dataDir string, logger *log.Logger) {
    store, _ := shepherd.NewStore(dataDir + "/shepherd.db")
    defer store.Close()

    stopCh := make(chan struct{})

    scheduler := shepherd.NewScheduler(store, logger)
    go scheduler.Run(stopCh)

    replicationCtrl := shepherd.NewReplicationController(
        store, scheduler, logger)
    go replicationCtrl.Run(stopCh)

    serviceCtrl := shepherd.NewServiceController(store, logger)
    go serviceCtrl.Run(stopCh)

    nodeCtrl := shepherd.NewNodeController(store, logger)
    go nodeCtrl.Run(stopCh)

    api := shepherd.NewAPIServer(addr, store, scheduler, logger)

    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)

    go func() {
        <-sigCh
        logger.Println("shutting down...")
        close(stopCh)
        api.Shutdown(context.Background())
    }()

    api.Start() // blocks here
}

graph TB
    MAIN["main goroutine<br/>api.Start() blocks"]

    subgraph "Background goroutines"
        G1["Scheduler<br/>every 2s"]
        G2["ReplicationController<br/>every 5s"]
        G3["ServiceController<br/>every 5s"]
        G4["NodeController<br/>every 10s"]
        G5["Signal handler"]
    end

    STOP["stopCh"] -.->|"close()"| G1 & G2 & G3 & G4

    STORE[("BoltDB Store")] <--> G1 & G2 & G3 & G4

    SIG["SIGTERM"] --> G5
    G5 -->|"close(stopCh)"| STOP
    G5 -->|"api.Shutdown()"| MAIN

All controllers share one store and one stopCh. Each one reads and writes the Store on its own without asking the others. The ServiceController builds endpoints, the NodeController watches heartbeats. How a signal turns into a clean stop is covered separately in Graceful Shutdown.

Agent: Two Tickers in One Select

The Agent has two jobs on different intervals. Instead of two goroutines, it uses one select over two channels:

func (a *Agent) Run(stopCh <-chan struct{}) error {
    a.mgr.Init()

    for attempt := 0; attempt < 10; attempt++ {
        if err := a.register(); err == nil {
            break
        }
        time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)
    }

    a.logger.Printf("agent %s started, reporting to %s",
        a.nodeName, a.apiAddr)

    heartbeat := time.NewTicker(10 * time.Second)
    defer heartbeat.Stop()

    reconcile := time.NewTicker(3 * time.Second)
    defer reconcile.Stop()

    for {
        select {
        case <-stopCh:
            a.logger.Println("agent stopped")
            return nil
        case <-heartbeat.C:
            a.sendHeartbeat()
        case <-reconcile.C:
            a.reconcilePods()
        }
    }
}

When several channels are ready at once, select picks one at random. So neither ticker starves: the heartbeat won't wait forever just because reconcile fires more often.

One problem, though: both branches run in the same goroutine, one after the other. If reconcilePods() starts a container and that takes 15 seconds, no heartbeat goes out for those 15 seconds. The NodeController on the server sees silence and may mark the node NotReady even though it's just busy. For Shepherd that's acceptable, but if reconcile ever gets heavy, the heartbeat belongs in its own goroutine.

Concurrent Access to the Store

Several goroutines read and write the Store at the same time. BoltDB gives you MVCC: many readers in parallel, one writer at a time. The Store adds a sync.RWMutex on top for Go-level operations:

type Store struct {
    db *bolt.DB
    mu sync.RWMutex

    podWatchers        []chan Event
    deploymentWatchers []chan Event
    watchMu            sync.Mutex
}

func (s *Store) ListPods(ns string) ([]*Pod, error) {
    s.mu.RLock()         // allows parallel reads
    defer s.mu.RUnlock()
    // ... s.db.View(...)
}

func (s *Store) CreatePod(pod *Pod) error {
    s.mu.Lock()          // exclusive access
    defer s.mu.Unlock()
    // ... s.db.Update(...)
}

func (s *Store) UpdatePod(pod *Pod) error {
    s.mu.Lock()
    defer s.mu.Unlock()

    key := nsKey(pod.Metadata.Namespace, pod.Metadata.Name)
    if err := s.put(bucketPods, key, pod); err != nil {
        return err
    }

    // Notify watchers (non-blocking)
    s.notify(s.podWatchers, Event{
        Type:   "Normal",
        Reason: "Updated",
        Object: "pod/" + pod.Metadata.Name,
    })
    return nil
}

To be fair, the RWMutex partly duplicates BoltDB, which already allows only one writer. What it adds is protection for things BoltDB doesn't see, like the watcher lists.

notify sends without blocking:

func (s *Store) notify(watchers []chan Event, evt Event) {
    s.watchMu.Lock()
    defer s.watchMu.Unlock()

    for _, ch := range watchers {
        select {
        case ch <- evt:
        default: // channel full - skip, don't block
        }
    }
}

default in a select means: if the channel is full (64-event buffer), drop the event. Losing an event is better than blocking a writer that holds s.mu, because while it waits, everyone waits. The reconciliation loop will pick up the lost change on the next tick anyway. The Event System follows the same "losing it is fine" rule.

Timeline of One Cycle

gantt
    title Goroutine ticks over the first 15 seconds
    dateFormat ss
    axisFormat %S

    section Scheduler (2s)
    tick :s1, 02, 1s
    tick :s2, 04, 1s
    tick :s3, 06, 1s
    tick :s4, 08, 1s
    tick :s5, 10, 1s
    tick :s6, 12, 1s
    tick :s7, 14, 1s

    section ReplicationCtrl (5s)
    tick :r1, 05, 1s
    tick :r2, 10, 1s
    tick :r3, 15, 1s

    section NodeCtrl (10s)
    tick :n1, 10, 1s

    section Agent heartbeat (10s)
    tick :h1, 10, 1s

    section Agent reconcile (3s)
    tick :a1, 03, 1s
    tick :a2, 06, 1s
    tick :a3, 09, 1s
    tick :a4, 12, 1s
    tick :a5, 15, 1s

The first tick doesn't arrive right away but after one full interval: time.NewTicker doesn't fire on creation. The Scheduler ticks most often, the NodeController least often. At second 10 four goroutines line up, and nothing bad happens, because each one works on its own.

Why Tickers Instead of Events

The event-driven approach: a pod changes, the scheduler gets notified. Faster, but more work:

  • An event gets lost. Now what?
  • A controller restarts and misses an event.
  • After a crash everything has to be reprocessed from scratch.

A ticker checks the whole state every N seconds. Slower, but:

  • A missed event breaks nothing, the next tick picks it up.
  • After a crash, state recovers by itself. It's just another tick.
  • No subscriptions, no replay, no queues.

In Kubernetes, informers combine both: watch for fast reaction, periodic resync for reliability. The kubelet does something similar, renewing its Lease roughly every 10 seconds whether or not anything changed. In Shepherd I kept tickers only. For dozens of resources that's enough, and there's a lot less code. I wrote about the broader idea in Desired vs Actual State and Async Scheduling.

Limitations

If reconcile takes longer than the interval, ticks don't pile up. time.Ticker holds at most one tick and silently drops the rest. The upside: after a long reconcile you don't get a burst of ten ticks in a row. The downside: a goroutine whose reconcile is always slow never rests, since a fresh tick is waiting the moment it finishes. With dozens of resources in Shepherd that's not an issue. With thousands you'd want rate limiting or a deduplicating queue like the workqueue in client-go.

A few more things that bite in practice:

  • A goroutine sleeping in time.Sleep during the Agent's register() retries ignores stopCh. Close the channel and it still finishes its nap. The last retry sleeps 5 seconds, and all ten add up to 27.5 seconds. That's a noticeable delay for graceful shutdown. The fix is a select on time.After and stopCh instead of Sleep.
  • recover only catches a panic in the goroutine where it happened. If reconcile() panics, the whole process goes down: Shepherd has no supervisor to restart a single controller. In a way that's honest, crashing and getting restarted by systemd beats limping along half-alive. But it also means one bug in the ServiceController takes the Scheduler down with it.
  • store, _ := shepherd.NewStore(...) ignores the error. If another process holds the DB file (BoltDB takes a flock on it), every controller gets nil and crashes on its first tick.

Fun Facts

  • The control loop is much older than Kubernetes. James Watt's centrifugal governor (1788) held a steam engine's speed steady: measure the speed, compare, close the valve a bit. In 1868 Maxwell wrote a paper about such devices, "On Governors", which is usually counted as the start of control theory.
  • "Kubernetes" is Greek for "helmsman". Norbert Wiener built the word "cybernetics" from the same root (kybernetes) in 1948 for the study of control and feedback. So the name literally describes what each of its controllers does.
  • A goroutine starts with a 2 KB stack, and has since Go 1.4. Go 1.2 used 8 KB, Go 1.0 used 4 KB, and stacks were made of segments. Segments caused the "hot split" problem: if a function in a loop was called right at a segment boundary, the runtime kept allocating and freeing a new chunk on every call. Go 1.3 switched to contiguous stacks that get copied when they grow.
  • The random choice in select is written into the language spec ("uniform pseudo-random selection"). It's not an implementation detail. Go randomizes on purpose so nobody can rely on branch order.
  • Go 1.23 changed timer channels: ticker.C used to have a buffer of 1, now it's synchronous. After Stop() or Reset() you can no longer read a stale tick from it. Most old code behaves the same, but len(ticker.C) and cap(ticker.C) now always return 0.
  • kube-controller-manager runs more than 30 controllers in one process, and each works the same way: its own goroutine, its own loop, shared state through the API. Shepherd does the same thing with four.

What I figured out while digging into this

At first I wanted to do it "properly": event-driven, watches, subscriptions. Then I noticed that half the code was handling "what if an event got lost". A ticker doesn't have that problem: the next tick arrives regardless. The "dumb" periodic reconcile is more reliable precisely because it's stateless, it doesn't care what happened between ticks. The better move here wasn't adding a mechanism, it was taking one away.

What could be improved

  • Replace the fixed interval with exponential backoff when there's nothing to do and a faster pace when there are Pending pods. Right now the scheduler ticks idly even on an empty cluster.
  • Add a sync.WaitGroup so main waits for the goroutines to actually finish after close(stopCh) instead of closing the channel and walking away.
  • Switch from stopCh to context.Context. Then deadlines and cancellation reach the reconcile functions themselves, not just the select loop.
  • Move the intervals into config. Right now 2s/5s/10s are hardcoded, and changing them means recompiling.
  • Wrap each reconcile() in a defer recover() with logging, so a panic in one controller doesn't take down the whole process.

Try It Yourself

# Start shepherd and watch the controller logs:
shepherd --mode standalone 2>&1 | grep -E 'started|reconcil'
# You'll see every controller ticking

How standalone differs from running server and agent separately is covered in Two-Mode Architecture.

Next up - CLI without frameworks: subcommand routing on plain os.Args.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Go Syscalls | Next: CLI Without Frameworks