Skip to content

Kubernetes vs Shepherd: A Minimal Orchestrator

Kubernetes vs Shepherd: A Minimal Orchestrator

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

Kubernetes consists of dozens of components. Shepherd implements five. Here's what turned out to be the bare minimum for a working orchestration system, and what we intentionally cut.

What Stayed

graph TB
    subgraph "Present in both"
        API["API Server<br/>K8s: kube-apiserver<br/>Shepherd: net/http REST"]
        STORE["State Store<br/>K8s: etcd cluster<br/>Shepherd: BoltDB file"]
        SCHED["Scheduler<br/>K8s: kube-scheduler<br/>Shepherd: filter + score"]
        RC["Replication Controller<br/>K8s: ReplicaSet controller<br/>Shepherd: reconcile loop"]
        SC["Service Controller<br/>K8s: endpoints controller<br/>Shepherd: label matching"]
        NC["Node Controller<br/>K8s: node lifecycle controller<br/>Shepherd: heartbeat timeout"]
        AGENT["Node Agent<br/>K8s: kubelet<br/>Shepherd: Agent"]
        CLI["CLI<br/>K8s: kubectl<br/>Shepherd: sheepctl"]
        EVENTS["Events<br/>K8s: events API<br/>Shepherd: events bucket"]
    end

That's 9 components. Kubernetes has 30+. The difference is what we cut.

What We Cut and Why

K8s Component What it does Why we cut it
etcd cluster Distributed consensus BoltDB is enough for single-node
Admission webhooks Validation/mutation on create Can validate in API handlers
RBAC Authorization Single user, no need
Service Accounts Pod identity No auth system
Secrets/ConfigMaps Configuration and secrets Pods take env directly
HPA Autoscaling Scale manually via sheepctl
CRDs Custom resources Fixed set of resources
CNI Network plugin interface One hardcoded bridge
CSI Storage plugin interface Volumes via bind mounts
kube-proxy Service routing (iptables) Endpoints as IP list
Ingress HTTP routing None
Watch API Long-poll for changes Ticker-based polling
Leader election HA controller manager Single instance
PodDisruptionBudget Controlled eviction No eviction
Rolling updates Zero-downtime deploys Delete + recreate

What We Learned: The Minimal Orchestrator

After building Shepherd, it became clear that container orchestration across multiple nodes needs exactly five things:

1. API Server - a single entry point. All components communicate through it. Without a central API, each component would need to know about every other one.

2. Persistent Store - the cluster state must survive restarts. Even an embedded DB is enough.

3. Scheduler - a node selection algorithm. Without it, you'd have to manually specify the node for every pod.

4. Controllers - reconciliation loops. Without them, "I want 3 replicas" doesn't work - you'd have to manually create each pod.

5. Agent - someone on the node has to turn "pod is assigned here" into actual running containers.

Everything else is a production extension: - Admission = input validation - RBAC = who can do what - CRD = extensibility - CNI/CSI = pluggable infrastructure - Watch = reactivity instead of polling

Comparison in Numbers

Metric Kubernetes Shepherd
Lines of code ~2,000,000 ~2,400
Go modules ~400 ~2 (bolt, unix)
Resource types 60+ 5 (Pod, Service, Deployment, Node, Event)
API endpoints 200+ ~15
Controllers 30+ 3
Min RAM 2GB+ ~20MB

Honest About Limitations

Shepherd is not production-ready:

  • Single point of failure - one API Server, one BoltDB file. Server goes down, the cluster is dead.
  • No TLS - all traffic is plain HTTP. Agent to API, sheepctl to API.
  • No auth - anyone can create/delete pods.
  • No rolling updates - scale down + scale up, with downtime.
  • No pod networking - containers in a pod don't share a network namespace.
  • No DNS - IP addresses only, no service discovery by name.
  • No health checks - liveness/readiness probes are missing.

But Shepherd shows that these 2400 lines are the skeleton on which Kubernetes's 2 million lines are built. All the same patterns: reconciliation, label selectors, async scheduling, desired state.

K3s - A Real Minimal Kubernetes

If you need a production-ready minimal orchestrator, check out K3s by Rancher. They did the same thing: replaced etcd with SQLite, bundled containerd, stripped cloud provider integrations. Single binary, ~100MB, full Kubernetes API compatibility.

The difference: K3s is production ready. Shepherd is educational.

Honest About Downsides

Without TLS and auth, anyone with network access can create and delete pods. That's acceptable for localhost, but for multi-node over a network it's a security hole. Adding TLS and auth means ~500 more lines of code and certificate management.

Two more things that are easy to underestimate:

  • Ticker polling instead of Watch scales badly. Every controller reads the entire state once per interval. Fine for dozens of resources, but at thousands the CPU burns re-reading objects that never changed.
  • BoltDB is single-writer. All writes serialize through one transaction. Fine for single-node, but it's a fundamental ceiling: it doesn't scale horizontally the way a Raft-based etcd cluster does.

💡 Fun facts

  • Kubernetes grew out of Borg, Google's internal orchestrator. The word "kubernetes" is Greek for "helmsman"; that's where the abbreviation "k8s" (8 letters between "k" and "s") and the ship's-wheel logo come from.
  • etcd is named the way it is on purpose: it's "/etc" + "distributed" - a distributed config store for the cluster. It uses Raft for consensus, which is exactly why Kubernetes recommends an odd number of nodes (3, 5, 7) by default - for quorum.
  • kube-scheduler actually runs in two phases, just like Shepherd: first filter (drop unfit nodes), then score (rank the rest). The same pattern we implemented in a few dozen lines.
  • Rancher's K3s stripped roughly a million lines of cloud-provider and legacy code from the binary, then wrapped the result so it still passes the full Kubernetes conformance test. "Minimal" doesn't mean "fake."

What I figured out while digging into this

My biggest takeaway was how much of Kubernetes's code goes not into orchestration but into extensibility. CRDs, admission webhooks, CNI, CSI - these are all mechanisms to let other people plug in without touching the core. The orchestration itself - reconcile loop + scheduler + store - is surprisingly small.

I expected the hard part to be scheduling or networking. It turned out the skeleton rests on a single idea: desired state vs actual state, and a loop that reconciles them. Everything else is plumbing around that one loop.

What could be improved

  • Replace ticker polling with a simple long-poll Watch endpoint - it's doable even on net/http and removes the main inefficiency of the controllers.
  • Add health checks (liveness/readiness): without them a Pod counts as alive until the process dies, even though the app inside may have been returning 500s for ages.
  • Extract the scheduler logic into a plugin interface (filter/score as separate functions) - a good exercise to feel how a real scheduler framework is built.
  • Add rolling updates to the ReplicationController instead of delete+recreate: scale up the new version, wait for Ready, then scale down the old one.

Try It Yourself

# Shepherd - one binary:
ls -lh ./shepherd        # ~10MB
# All state - one file:
ls -lh /var/lib/shepherd/shepherd.db
# Compare with K8s:
# etcd cluster + kube-apiserver + kube-scheduler + ...

Next up - where containers, images, and cluster state physically live on disk.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Docker vs Sheep | Next: Filesystem Layout