Two-Mode Architecture: Server + Agent

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

---

Shepherd is one binary, three modes. --mode server starts the control plane. --mode agent starts a worker. --mode standalone starts everything together. Same code, different configuration.

Everything the previous parts built separately - the API Server, the BoltDB store, the scheduler, the reconciliation loops, the node agent - meets in one main() here. The only question left is which of them a given process actually starts.

Entry point

func main() {
    var (
        addr     = flag.String("addr", ":9876",
            "API server listen address")
        dataDir  = flag.String("data-dir",
            "/var/lib/shepherd", "Data directory")
        nodeName = flag.String("node-name", "",
            "Node name (for agent mode)")
        apiAddr  = flag.String("api-addr", "",
            "API server address (for agent mode)")
        mode     = flag.String("mode", "server",
            "Run mode: server, agent, or standalone")
    )
    flag.Parse()

    logger := log.New(os.Stdout, "[shepherd] ",
        log.LstdFlags|log.Lshortfile)

    switch *mode {
    case "server":
        runServer(*addr, *dataDir, logger)
    case "agent":
        if *apiAddr == "" {
            fmt.Fprintln(os.Stderr,
                "agent mode requires --api-addr")
            os.Exit(1)
        }
        runAgent(*nodeName, *apiAddr, logger)
    case "standalone":
        runStandalone(*addr, *dataDir, *nodeName, logger)
    }
}

What each mode starts

graph TB
    subgraph "server mode"
        S_API["API Server :9876"]
        S_SCHED["Scheduler (goroutine)"]
        S_RC["ReplicationController (goroutine)"]
        S_SC["ServiceController (goroutine)"]
        S_NC["NodeController (goroutine)"]
        S_DB[("BoltDB")]
        S_API --> S_DB
        S_SCHED --> S_DB
        S_RC --> S_DB
    end

    subgraph "agent mode"
        A_AGENT["Agent"]
        A_SHEEP["Sheep Runtime"]
        A_AGENT -->|"HTTP"| REMOTE["Remote API Server"]
        A_AGENT --> A_SHEEP
    end

    subgraph "standalone mode"
        ST_API["API Server :9876"]
        ST_SCHED["Scheduler"]
        ST_RC["ReplicationController"]
        ST_SC["ServiceController"]
        ST_NC["NodeController"]
        ST_DB[("BoltDB")]
        ST_AGENT["Agent"]
        ST_SHEEP["Sheep Runtime"]
        ST_AGENT -->|"HTTP localhost:9876"| ST_API
        ST_AGENT --> ST_SHEEP
    end

Server mode

func runServer(addr, dataDir string, logger *log.Logger) {
    os.MkdirAll(dataDir, 0755)

    store, _ := shepherd.NewStore(dataDir + "/shepherd.db")
    defer store.Close()

    stopCh := make(chan struct{})

    scheduler := shepherd.NewScheduler(store, logger)
    go scheduler.Run(stopCh)

    replicationCtrl := shepherd.NewReplicationController(
        store, scheduler, logger)
    go replicationCtrl.Run(stopCh)

    serviceCtrl := shepherd.NewServiceController(store, logger)
    go serviceCtrl.Run(stopCh)

    nodeCtrl := shepherd.NewNodeController(store, logger)
    go nodeCtrl.Run(stopCh)

    api := shepherd.NewAPIServer(addr, store, scheduler, logger)

    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
    go func() {
        <-sigCh
        close(stopCh)
        api.Shutdown(context.Background())
    }()

    api.Start()
}

5 goroutines: scheduler, 3 controllers (replication, service, node), and the signal handler from the previous part. The API Server blocks the main goroutine - that's the only thing keeping main alive.

Agent mode

func runAgent(nodeName, apiAddr string, logger *log.Logger) {
    agent := shepherd.NewAgent(nodeName, apiAddr, logger)

    stopCh := make(chan struct{})
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
    go func() { <-sigCh; close(stopCh) }()

    if err := agent.Run(stopCh); err != nil {
        logger.Fatalf("agent: %v", err)
    }
}

The Agent communicates with the API Server only via HTTP. It doesn't import the scheduler or controller packages. This means server and agent can run on different machines, and it means the import graph, not a code comment, is what enforces the boundary. If someone ever wires a direct call in, the compiler starts pulling BoltDB into the agent binary, and you notice.

Standalone mode: the interesting part

func runStandalone(addr, dataDir, nodeName string,
    logger *log.Logger) {
    os.MkdirAll(dataDir, 0755)

    store, _ := shepherd.NewStore(dataDir + "/shepherd.db")
    defer store.Close()

    stopCh := make(chan struct{})

    // Start everything like in server mode
    scheduler := shepherd.NewScheduler(store, logger)
    go scheduler.Run(stopCh)
    // ... controllers ...

    api := shepherd.NewAPIServer(addr, store, scheduler, logger)

    // API Server in a goroutine (don't block main)
    go func() {
        if err := api.Start(); err != nil &&
            err.Error() != "http: Server closed" {
            logger.Fatalf("api server: %v", err)
        }
    }()

    // Agent connects to localhost
    if nodeName == "" {
        host, _ := os.Hostname()
        nodeName = host
    }
    actualAddr := addr
    if actualAddr[0] == ':' {
        actualAddr = "localhost" + actualAddr
    }
    agent := shepherd.NewAgent(nodeName, actualAddr, logger)

    go func() {
        agent.Run(stopCh) // Agent retry handles API readiness
    }()

    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM)
    <-sigCh

    close(stopCh)
    api.Shutdown(context.Background())
}

Here's the interesting bit: the Agent connects to the API Server via HTTP even when they're in the same process. That's why the Agent has retry on registration - the API Server might not be ready yet:

for attempt := 0; attempt < 10; attempt++ {
    if err := a.register(); err == nil {
        break
    }
    a.logger.Printf("agent: register attempt %d failed, retrying...",
        attempt+1)
    time.Sleep(time.Duration(attempt+1) * 500 * time.Millisecond)
}

Backoff of 500ms, 1s, 1.5s, 2s... Maximum 10 attempts. Strictly speaking that's linear backoff, not exponential - the delay grows by a constant, not by a factor. It works here because the wait is bounded and there's exactly one client; with fifty agents reconnecting to a restarted server at once, this is precisely the pattern that produces a thundering herd.

The same heartbeat and registration machinery runs in every mode. Standalone just makes the race visible on the very first start.

Why HTTP even for localhost

You could make direct Go function calls in standalone mode. But then: - You'd need a separate code path for standalone - Bugs might only show up in multi-node but not in standalone - Agent code gets more complex (two interfaces: HTTP and direct)

HTTP always - one code path. Standalone works the same as multi-node, just localhost instead of a remote address.

It's the same argument as desired vs actual state: a component that has to be correct only against an interface, not against a deployment topology, is much harder to break. And since pod creation was already asynchronous, nothing in the flow depended on the agent and the API Server being co-located anyway.

Deployment topologies

graph TB
    subgraph "Dev/Test: standalone"
        DEV["shepherd --mode standalone"]
        DEV --> DEV_C["Containers on the same machine"]
    end

    subgraph "Small: 1 server + N agents"
        SMALL_S["shepherd --mode server<br/>machine-1"]
        SMALL_A1["shepherd --mode agent<br/>machine-2"]
        SMALL_A2["shepherd --mode agent<br/>machine-3"]
        SMALL_A1 -->|HTTP| SMALL_S
        SMALL_A2 -->|HTTP| SMALL_S
    end

Where you can trip up

Standalone mode has a single point of failure: one process goes down - the entire cluster is dead. In multi-node mode, the agent keeps running even if the server goes down (containers don't stop, only new pods can't be created). That's not resilience by design, it's a side effect of the agent owning its own reconcile loop, but it's real, and it's exactly how a kubelet survives a control plane outage.

Second problem: the standalone agent and server share one machine. If a container eats all the memory, it kills the control plane too. In multi-node mode, the control plane is isolated.

Two more small things that are easy to trip over: - In standalone, the agent connects to localhost:9876. If someone overrides --addr to a specific interface (not :9876), the actualAddr[0] == ':' parsing breaks and the agent dials the wrong place. - Registration retry is capped at 10 attempts (~27.5s total). If the API Server takes longer to start, the agent gives up and the node never registers. - actualAddr[0] indexes into a string without checking that it's non-empty - --addr "" panics before the agent ever dials anything. - The switch has no default. --mode serevr doesn't fail: the process starts, prints nothing, and exits 0. A typo looks exactly like a successful run.

💡 Fun facts

• Kubernetes is also essentially "one codebase, different roles," but it ships them as separate binaries: kube-apiserver, kube-scheduler, kube-controller-manager, kubelet. K3s from Rancher went the other way - it crammed everything into a single k3s binary with server and agent subcommands, exactly like we do.
• The kubelet (Kubernetes' agent) also talks to the API Server only over HTTP/gRPC, even when they're on the same machine. No direct calls - the same principle as here.
• "Standalone" in our sense is basically what Docker Desktop and minikube do for a developer: control plane and worker in one process/VM, so you don't have to stand up a cluster for a single container.
• The "one binary, mode picked by a flag" trick is an old unix tradition: busybox decides who to be based on argv[0]. We do the same thing, just with an explicit --mode flag.
• Docker went the opposite way over the same years. The single docker daemon of 2013 was pulled apart into dockerd, containerd, containerd-shim, and runc - precisely so that restarting the daemon wouldn't kill running containers. Consolidation and decomposition are both right answers; they just optimize for different things (one-command install vs. independent restartability).
• k0s packs even more into one file than k3s: containerd, runc, and the whole control plane ship inside a single static binary, so k0s controller on a bare VM needs no container runtime installed at all.
• Kubernetes has its own "standalone" trick, and it's stranger than a flag: the control plane components run as static pods - YAML files in /etc/kubernetes/manifests that the kubelet starts directly, without asking any API Server. The kubelet boots the API Server that the kubelet then registers with. That circular dependency is what kubeadm quietly resolves for you.
• Loopback HTTP isn't as expensive as it sounds: traffic over lo never touches a NIC or driver, and on Linux the loopback MTU is 65536 bytes instead of 1500, so a large JSON body crosses the kernel in a handful of segments rather than dozens.
• Go's standard flag package has no notion of subcommands and no enum type, which is exactly why so many CLI tools reach for Cobra, and why --mode server here is a flag rather than a shepherd server subcommand. Since Go 1.16 flag.Func lets you validate a flag's value at parse time, which would turn our silent typo into an error for free.
• "Same binary, different role" also shows up where you'd least expect it: systemd is systemd as PID 1 and systemd --user per session, and git is a single dispatcher that execs git-<subcommand> from its libexec directory.

What I figured out while digging into this

At first I wanted to make standalone "smart" - to have the agent in the same process call the control plane directly, without HTTP. It seemed faster and cleaner. It turned out the opposite: it would have introduced a second code path that nobody would really test, and the bugs would have lived right there.

Once I forced myself to go through HTTP even on localhost, everything suddenly simplified - standalone became just a special case of multi-node. And it clicked that the registration retry isn't a "standalone hack" but an honest reflection of the fact that in a distributed system nobody guarantees startup order.

What could be improved

• Replace the fixed retry (10 attempts) with a real API Server health-check endpoint: the agent waits for GET /healthz instead of guessing by a timer.
• Add --mode validation: right now an unknown mode silently does nothing (switch with no default). It should fail with an explicit error.
• Make graceful shutdown symmetric for standalone: first wait for the agent to stop, then the API Server, so pods get a chance to terminate cleanly.
• For multi-node, add TLS and authentication between agent and server - right now HTTP is wide open, which is only acceptable for dev.
• Use flag.Func for --mode so an unknown value fails at parse time with the list of valid modes, instead of exiting 0 in silence.
• Parse --addr with net.SplitHostPort instead of actualAddr[0] == ':' - it handles the empty host, IPv6 literals, and an empty string without panicking.

Try it yourself

# Standalone (all-in-one):
sudo shepherd --mode standalone --addr :9876
# In another terminal:
sheepctl info
sheepctl nodes

# Multi-node:
# Terminal 1 (control plane):
shepherd --mode server --addr :9876 --data-dir /tmp/shepherd-server
# Terminal 2 (worker 1):
sudo shepherd --mode agent --node-name worker-1 --api-addr localhost:9876
# Terminal 3 (worker 2):
sudo shepherd --mode agent --node-name worker-2 --api-addr localhost:9876
# Terminal 4:
sheepctl nodes  # you'll see worker-1 and worker-2

---

Next up - BoltDB vs etcd: when an embedded database is enough and when you need distributed consensus.

Resources

• Kubernetes Components: control plane vs node components
• Kubernetes architecture: full architectural overview
• kubelet: the agent our agent mode is modelled on
• Static pods: how the kubelet bootstraps a control plane with no API Server
• k3s architecture: server and agent subcommands of one binary
• k0s: single static binary with containerd and runc inside
• BusyBox FAQ: one binary, many personalities via argv[0]
• flag and flag.Func - stdlib flags and parse-time validation
• net.SplitHostPort: the right way to take apart a listen address
• Exponential Backoff and Jitter: why linear retry across many clients ends badly

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Graceful Shutdown | Next: Embedded vs External DB