Skip to content

Graceful Shutdown: Signals, Channels, Cleanup

Graceful Shutdown: Signals, Channels, Cleanup

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

Shepherd runs several goroutines: scheduler, replication controller, service controller, node controller, API server, agent. When SIGTERM arrives, they all need to stop cleanly. Here's how to do it with a single channel.

Stop channel

func runServer(addr, dataDir string, logger *log.Logger) {
    store, _ := shepherd.NewStore(dataDir + "/shepherd.db")
    defer store.Close()

    stopCh := make(chan struct{})

    scheduler := shepherd.NewScheduler(store, logger)
    go scheduler.Run(stopCh)

    replicationCtrl := shepherd.NewReplicationController(
        store, scheduler, logger)
    go replicationCtrl.Run(stopCh)

    serviceCtrl := shepherd.NewServiceController(store, logger)
    go serviceCtrl.Run(stopCh)

    nodeCtrl := shepherd.NewNodeController(store, logger)
    go nodeCtrl.Run(stopCh)

    api := shepherd.NewAPIServer(addr, store, scheduler, logger)

    // Graceful shutdown
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)

    go func() {
        <-sigCh
        logger.Println("shutting down...")
        close(stopCh)
        api.Shutdown(context.Background())
    }()

    api.Start()
}

graph TB
    SIG["SIGTERM / SIGINT"] --> CLOSE["close(stopCh)"]
    CLOSE --> S["Scheduler: return"]
    CLOSE --> RC["ReplicationController: return"]
    CLOSE --> SC["ServiceController: return"]
    CLOSE --> NC["NodeController: return"]
    CLOSE --> API["api.Shutdown()"]

A single close(stopCh) stops all goroutines. Each controller has a select on stopCh:

func (s *Scheduler) Run(stopCh <-chan struct{}) {
    ticker := time.NewTicker(2 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-stopCh:
            s.logger.Println("scheduler stopped")
            return
        case <-ticker.C:
            s.reconcile()
        }
    }
}

When the channel is closed, <-stopCh returns the zero value without blocking. All goroutines listening on this channel wake up and terminate.

Why close() instead of send?

close(ch) wakes all listeners. ch <- struct{}{} wakes only one. With close, a single line stops everything.

Standalone mode

In standalone mode, the API Server and Agent run in the same process:

func runStandalone(addr, dataDir, nodeName string,
    logger *log.Logger) {
    // ... start all controllers ...

    go api.Start()
    go agent.Run(stopCh)

    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
    <-sigCh

    logger.Println("shutting down...")
    close(stopCh)
    api.Shutdown(context.Background())
}

<-sigCh blocks the main goroutine until a signal arrives. After the signal, close(stopCh) stops everything.

Why this is better than context.Cancel

You could use context.WithCancel. But for simple cases, a channel is more straightforward: one close - everything stops. Context adds complexity (Done(), Err(), Value()) that isn't needed here.

Something to keep in mind

We don't wait for reconcile to finish. close(stopCh) signals a stop, but if a controller is in the middle of a write operation to BoltDB, the transaction may not complete cleanly. In production you'd need a graceful drain with a timeout.

Two separate traps. close(stopCh) twice panics - closing an already-closed channel crashes the process; so exactly one owner should ever close it. And api.Shutdown(context.Background()) without a deadline can hang forever if a long-lived keep-alive connection remains: a context with no timeout here means "wait forever."

💡 Fun facts

  • close(ch) as a broadcast is an idiom described back in Rob Pike's "Go Concurrency Patterns." A closed channel hands out the zero value an unlimited number of times, so all listeners wake up at once. You can't do that with a send.
  • SIGKILL (kill -9) can't be caught or ignored - the kernel kills the process with no chance for cleanup. That's why graceful shutdown reacts to SIGTERM: it's a "polite" request you can still respond to. SIGSTOP is the second signal you can't trap - it freezes the process without asking.
  • Kubernetes gives a pod terminationGracePeriodSeconds (30 by default): first SIGTERM, then a wait, and only then SIGKILL. Our close(stopCh) is that same "polite" first step, just without the timer.
  • docker stop follows the same script but with a 10-second default (--time changes it): SIGTERM, wait, SIGKILL. So the timeout in your close(stopCh) should be smaller than the orchestrator's, or you'll always eat the SIGKILL.
  • net/http has had a built-in Server.Shutdown(ctx) since Go 1.8: it stops accepting new connections and waits for in-flight ones to finish. That's exactly what we call - no need to reinvent it.
  • PID 1 is special: the kernel doesn't apply default signal actions to it, so a process running as PID 1 in a container that doesn't explicitly handle SIGTERM simply ignores it, and docker stop waits out the full grace period before SIGKILL. That's the whole reason tini and --init exist.
  • Since Go 1.16 there's signal.NotifyContext: it folds the signal handler and a context.Context into one object, so the arrival of SIGTERM cancels the context and the manual sigCh disappears entirely.
  • A chan struct{} is a zero-byte type: the value carries no data, only the event (the close) matters. It's the canonical "signal-only" channel, and the compiler even avoids allocating for its elements.

What I figured out while digging into this

For a long time I was puzzled why everyone recommends a chan struct{} rather than a chan bool - until I wrote this shutdown myself. struct{} takes zero bytes and says clearly: the value carries no meaning, only the fact of the event matters (here, the close). But the bigger realization is more down-to-earth: I was used to thinking of stopping as "send a stop command." It turns out the cleaner model is to send nothing at all - just close the channel and let each goroutine notice it in its own select. Less code, and no message for anyone to lose.

What could be improved

  • Add a sync.WaitGroup: the shutdown goroutine should wait until all controllers have actually exited before closing the store and the process.
  • Give api.Shutdown a context with a timeout (context.WithTimeout), so a keep-alive hang doesn't block the exit forever.
  • Implement a real drain: a controller in the middle of a reconcile should finish the current iteration (or roll back the transaction) rather than getting cut off mid-write.
  • Move signal handling to signal.NotifyContext (Go 1.16+) - it ties the signal and the context into one object and removes the manual sigCh.
  • Consider golang.org/x/sync/errgroup when goroutines can fail: it combines a WaitGroup, error propagation, and a shared context cancel in one primitive.

Try it yourself

# Start shepherd and stop with Ctrl+C:
sudo ./shepherd --mode standalone
# You'll see in the logs:
# shutting down...
# scheduler stopped
# replication controller stopped

Next up - Two-Mode Architecture: one binary for server, agent, and standalone.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Async Scheduling | Next: Two-Mode Architecture