Blog¶

Welcome to my blog! Here you'll find articles about DevOps, AWS, cloud architecture, and more.

Recent Posts¶

Filesystem Layout: Where Containers Live ¶

November 7, 2026

Everything in Sheep is files: container state, images, overlay layers, the IP counter. No database and no daemon holding state in memory. Restart Sheep and it reads the tree under /var/lib/sheep and knows everything that happened before.

Four directories created on first run: containers/ with one state.json per container, overlay/ with upper/work/merged per container, images/ with manifest.json plus a read-only rootfs/, and network/ with a single ip_counter file.
The file lifecycle, step by step: create makes the directories and mounts overlay, start and stop only rewrite state.json, rm unmounts and deletes.
Copy-on-write in practice: ten nginx containers share one read-only rootfs, and each writes only into its own upper/. The kernel shows the union in merged/, which becomes the container's /.
Shepherd keeps the whole cluster (pods, services, deployments, nodes, events) in one BoltDB file. Back it up with cp.
Against /var/lib/docker: Docker stores each layer separately in a content-addressable store, with link files and short-name symlinks. Sheep keeps a full rootfs per image, which is simpler but has no deduplication.
What is not ideal: os.WriteFile is not atomic, so a crash mid-write can corrupt state.json. Overlay mounts orphaned by a crash linger in /proc/mounts. Ten images on one Ubuntu base mean ten copies of it.
Fixes worth making: temp file plus rename for state, content-addressable layers, GC for orphaned overlays on startup, and mounting merged only while a container is running.
Side notes: OverlayFS needs upper and work on the same filesystem and an empty work; Docker's l/ symlinks exist because mount has a limit on argument length; /var/lib comes from the FHS; runc keeps its own state.json per container too.

Part thirty-eight of the Sheep & Shepherd series.

Kubernetes vs Shepherd: A Minimal Orchestrator ¶

November 1, 2026

Kubernetes is 30+ components and about two million lines of code. Shepherd runs a multi-node cluster in about 2,400 lines, and building it shows which parts orchestration actually needs and which parts are there for production.

The minimum is five things: an API server as the single entry point, a persistent store, a scheduler, controllers running reconcile loops, and an agent on each node that turns "pod assigned here" into running containers.
What got cut, and why: an etcd cluster (BoltDB is enough on one node), admission webhooks, RBAC, Secrets/ConfigMaps, HPA, CRDs, CNI/CSI, kube-proxy, Ingress, the Watch API, leader election, and rolling updates.
Most of Kubernetes's code is not orchestration but extensibility: CRDs, webhooks, CNI and CSI exist so other people can plug in without touching the core.
The numbers: ~2,400 lines against ~2,000,000, 5 resource types against 60+, ~15 API endpoints against 200+, ~20 MB of RAM against 2 GB+.
The known limits: a single point of failure, no TLS, no auth, no DNS, no health checks, and downtime on every update. Ticker polling re-reads the whole state on every interval, and BoltDB pushes every write through a single writer.
K3s is the production version of the same idea: SQLite instead of etcd, containerd bundled in, and it still passes the Kubernetes conformance tests.

Part thirty-seven of the Sheep & Shepherd series.

Agents & Dragons: Once Upon Agentic AI Workshop ¶

October 26, 2026

Monday, 26 October, at the AWS UG Meetup in Warsaw: the Agents & Dragons workshop, where tool calling is a spell and the agent runs a dwarf named Thorin through a quest.

Not a slide deck. A comic and D&D: @tool in Python becomes a spell, and the Game Master breaks a stealth check down to the dice.
Familiar pieces (tools, MCP, agent-to-agent) land differently when you have to use them in a quest.
Thanks to Michał Krzyżanowski for the format, and to Janusz Moroz, Luke Dorosz, and AWS for having us.

Agentic Football Cup at AWS UG Kyiv ¶

October 23, 2026

Friday, 23 October, at AWS UG Kyiv: five agents on Amazon Bedrock play football. I-go-go had 65% of the ball and zero shots.

On nova-pro the defender took 1.2-1.9 s. Press and Intercept instead of Shoot. Four losses, 0-7.
On nova-lite answers came back in about 110 ms, and there were two 1-0 wins. But 77% possession and 75 Shoot commands still meant zero shots on target.
We reached the third-place match and lost it. Bronze went to SonicCrusaders, the tournament to Karpaty Lviv.

Docker vs Sheep: What a Container Runtime Actually Does ¶

October 11, 2026

Docker is five components stacked on top of each other - docker CLI, dockerd, containerd, containerd-shim, runc. Sheep is one binary that does the same job, and the comparison shows exactly what each of those layers is actually buying.

Every layer answers a specific production problem: dockerd is the API surface, containerd is lifecycle and images, the shim is one process per container holding its file descriptors, and runc is a one-shot tool that creates namespaces and exits.
Sheep collapses all of it: the CLI calls the Manager directly, the Manager creates namespaces directly. No daemon, no gRPC, no shim.
Component by component, what gets lost: remote control, surviving a daemon restart, interchangeable runtimes, layer deduplication, storage backend choice, port mapping, docker build.
What both do identically at the kernel level: clone() with CLONE_NEW*, pivot_root(), proc/sys/dev mounts, cgroups v2, bridge plus veth, iptables MASQUERADE. That is the whole container. Everything above it is UX, reliability, and ecosystem.
The shim gap, spelled out. Kill Sheep with ten containers running and the orphans keep going, but on restart Sheep only has the PID to go on, so the exit code is gone. Docker keeps a shim alive per container precisely for that.
The numbers: ~8 MB against ~50 MB, ~2,500 lines against 100,000+, 5 dependencies against ~200, 10 ms startup against ~2 s of daemon.
Where the monolith bites: any bug in the manager kills the whole runtime, upgrading Sheep stops every container because there is no data plane / control plane split, and one rootfs per image means ten nginx containers hold ten copies of the filesystem.
Side notes: containerd was deliberately spun out to the CNCF so Kubernetes could use it without Docker, which is how dockershim removal eventually happened; runc is Docker's own libcontainer extracted into a binary; the shim exists mostly to solve reparenting; and "container" is not a kernel object at all, just a user-space convention over namespaces, cgroups, and processes.

Part thirty-six of the Sheep & Shepherd series.

JSON Serialization for State Persistence ¶

October 4, 2026

One format carries everything in Sheep and Shepherd: state.json per container, manifest.json per image, BoltDB values stored as JSON bytes, the whole HTTP API, and the OCI manifests on top.

Container state goes to disk through MarshalIndent with two spaces, so you can cat it and read it. Image manifests use plain Marshal, because nobody reads those.
Struct tags do the mapping: snake_case on the wire, CamelCase in Go, and time.Time becomes RFC3339 for free.
Loading on startup is deliberately defensive. A corrupted state.json is skipped rather than fatal, and a container that claims to be running with a dead PID gets rewritten to stopped.
The API streams both ways, json.NewDecoder(r.Body) in and json.NewEncoder(w) out, so nothing is buffered whole.
Against protobuf: implicit schema instead of a .proto file, 30-50% more bytes, 2-5x slower. Why stdlib still wins at this scale, and why Kubernetes serves JSON but stores protobuf.
The sharp edges, spelled out. omitempty cannot tell "zero" from "unset", which is why Kubernetes APIs are full of *int32. Unmarshal silently swallows unknown fields and hides schema drift. There is no schema version anywhere. And a crash mid-WriteFile leaves half-written JSON that an atomic rename would have prevented.
Side notes: why Crockford left comments out of JSON, Go's deterministically sorted map keys, and []byte quietly becoming base64.

Part thirty-five of the Sheep & Shepherd series.

CLI Without Frameworks: Subcommand Routing ¶

October 2, 2026

Sheep and sheepctl carry no cobra, no urfave/cli, nothing: just os.Args and the standard flag package, around 1170 lines across two binaries and 30+ commands.

Routing is two switches. The first handles commands that need no Manager (init, version, help, where init is the re-exec from inside the namespace), the second handles everything else.
parseRunFlags walks os.Args[2:] by hand: -- flags take the next argument, -d style booleans take none, the first bare token becomes the image and the rest becomes the command. From the outside it feels exactly like docker run.
Sheepctl mixes the two approaches: flag.Parse() for the global -api and -n, then flag.Args() into a manual switch for subcommands.
Output goes through a small homegrown cli.Table that tracks column widths as rows are added.
Against cobra: zero dependencies and about 100 lines of routing, versus roughly 3 MB of binary, shell autocompletion and generated help.
What the manual parser leaves broken: --typo quietly becomes the image name, --name at the end of the line disappears because i++ runs off the slice, and --api in one binary sits next to -api in the other.
Side notes: why the flag package treats -n and --n as the same thing, how Docker itself eventually moved to cobra, and what BusyBox does with os.Args[0].

Part thirty-four of the Sheep & Shepherd series.

Goroutines for Control Loops ¶

September 26, 2026

Shepherd keeps 6 goroutines running, each on its own clock: the Scheduler every 2 seconds, the ReplicationController every 5, the Agent sending a heartbeat every 10 and checking pods every 3. They never talk to each other. All they share is the BoltDB Store and one stopCh.

Every controller is the same three lines of structure: a time.NewTicker, a select over two channels, and a close(stopCh) that every receiver sees at once.
The Store sits behind a sync.RWMutex. Watchers get events through a non-blocking send, so a full 64-event channel is simply skipped, because the next tick catches up anyway.
Tickers instead of events, on purpose: a lost event breaks nothing, and crash recovery is just another tick.
Three things that are wrong with it. The Agent's heartbeat and reconcile share one goroutine, so a slow container start can get the node marked NotReady. The time.Sleep retry loop never hears stopCh and can hold shutdown for up to 27.5 seconds. A panic in one controller takes the whole process down.
Some history on the way: Watt's 1788 governor, Maxwell's "On Governors", the Greek root shared by "Kubernetes" and "cybernetics", and what exactly Go 1.23 changed about tickers.

Part thirty-three of the Sheep & Shepherd series.

Go's syscall Package: mount, clone, pivot_root ¶

September 19, 2026

Everything the runtime does bottoms out in a handful of system calls, and Go reaches them through two packages: the stdlib syscall, frozen since Go 1.4, and golang.org/x/sys/unix, where anything newer lives. Mount(), Unmount() and Sethostname() come from the first. PivotRoot(), Mknod() and Mkdev() exist only in the second.

clone() is never exposed at all. The Go runtime needs control over thread creation, so the namespace flags ride along in SysProcAttr.Cloneflags during fork+exec, with Unshareflags: CLONE_NEWNS making mount propagation private so nothing leaks back to the host.
Six mounts build the container: proc, sysfs, a tmpfs /tmp, a /dev tmpfs locked down with MS_NOSUID|MS_STRICTATIME, the overlay with its lowerdir/upperdir/workdir option string, and a self-bind with MS_BIND|MS_REC that pivot_root insists on.
Then the root swap itself, with the old root parked in .pivot_old and detached lazily through MNT_DETACH.
Five device nodes are built by hand from Mkdev(major, minor) plus Mknod(path, S_IFCHR|mode, dev), with /dev/fd, stdin, stdout and stderr as symlinks into /proc/self/fd.
Signals turn out to be the entire process-control API: SIGTERM, then SIGKILL, and signal 0 as a free liveness probe. The kernel answers ESRCH when the PID is gone, which is how a stale "running" in state.json gets corrected on load.
Cgroups are not syscalls at all. os.WriteFile() into /sys/fs/cgroup is plain open/write/close.
Where it falls short: networking shells out to ip, nsenter and iptables instead of netlink, so it is slower and depends on what the host has installed. A mount namespace is bound to an OS thread, so anything entering one belongs under runtime.LockOSThread() or Go may reschedule the goroutine onto a thread that no longer exists. And syscall.Errno wrapped in a bare %w throws away the difference between EPERM and EINVAL.
Side notes: chroot only changes / for a process while pivot_root physically swaps the root mount point, which is why runtimes pick the latter; clone() carries over 20 flags and is one of the most complex calls in Linux; and /dev/null has been major 1, minor 3 on every Linux machine in the world since the device registry was fixed in the 1970s.

Part thirty-two of the Sheep & Shepherd series.

Build Tags: One Codebase for Linux and macOS ¶

September 11, 2026

Sheep stands on namespaces, cgroups, overlay mounts and bridge networking. None of that exists on macOS, and the thing still has to compile and run on a Mac.

runtime.GOOS == "linux" inside an if does not help: syscall.CLONE_NEWPID is not a symbol on darwin at all, so the compiler fails before any branch could run.
The split happens at file level instead. runtime_linux.go behind //go:build linux holds namespaces, cgroups, overlay and device nodes; runtime_stub.go behind //go:build !linux holds a bare exec.Command. Same trick again for network_linux.go and network_stub.go.
Everything above them, container.go, manager.go, image.go, carries no tag at all. That is the real payoff: the Manager coordinating the container lifecycle calls startContainer() and never learns which platform it got.
The overlay fallback falls out of the same design. On macOS mountOverlay() from the stub returns an error and the Manager quietly copies the rootfs instead, so the container still runs, just slower and without copy-on-write.
What this costs: demo mode on macOS exercises no real isolation, so code that works on a Mac can still die on Linux namespace quirks and only Linux CI will catch it. The networking stub is a silent no-op that never says "you are not isolated". And there is no shared runtime.go contract, so the stub and the Linux version can drift apart unnoticed.
Side notes: //go:build replaced // +build in Go 1.17 precisely because a stray space could break the old syntax; the filename suffix is itself a build constraint, so foo_linux.go never enters a macOS build no matter what is inside it; a missing blank line between the tag and package turns the constraint into an ordinary comment; and Docker, containerd and Kubernetes all rest on the same pile of *_linux.go / *_windows.go / *_unsupported.go files.

Opens the Go Systems Programming block. Part thirty-one of the Sheep & Shepherd series.

Embedded vs External DB: BoltDB vs etcd Trade-offs ¶

September 5, 2026

Kubernetes stores cluster state in etcd: three to five nodes, Raft consensus, real infrastructure. Shepherd stores its state in BoltDB: one file, a single bolt.Open() call, nothing to deploy. Both are key-value stores, and the difference only shows up when something breaks.

The comparison runs across deployment, size, transactions, replication, watch and backup, but watch is where the real gap sits. Ours is an in-memory Go channel with a 64-slot buffer and a select/default, so a restart drops every subscription and a burst silently drops events. etcd's watch carries a revision, so a reconnecting client catches up on everything it missed.
That one difference is why the controllers poll on a ticker and treat watch as an accelerator rather than a source of truth. It reads as an honest admission of eventual consistency rather than a backward design.
BoltDB's own limits: exactly one writer, so a long Update() blocks every other write; bolt.Open() takes an exclusive file lock, so pointing standalone and server at the same --data-dir hangs the second one; a cp of a hot file can be inconsistent, and the right answer is Tx.WriteTo() inside a read transaction; the file never shrinks on its own; and a forgotten long-running View() pins pages and bloats it.
etcd is not free either. It needs quorum, does two disk syncs per write, and can be more complex than the cluster it serves.
Side notes: etcd itself uses bbolt as its storage engine, so every etcd node is that same BoltDB with Raft, MVCC and gRPC layered on top. The original boltdb/bolt was frozen by Ben Johnson and the world moved to the etcd team's fork. An even-sized etcd cluster tolerates no more failures than the odd one below it. The default 2 GB backend quota flips a cluster into a read-only NOSPACE alarm that needs a manual disarm. ErrCompacted means even guaranteed watch delivery is only guaranteed within a retention window. The two meta pages at the head of a .db file are what make a crash mid-commit survivable. And k3s ships kine, an etcd API that speaks to SQLite, Postgres or MySQL, so Kubernetes can run with no etcd at all.

Part thirty of the Sheep & Shepherd series.

Two-Mode Architecture: Server + Agent ¶

August 30, 2026

One binary, three modes. --mode server starts the control plane, --mode agent starts a worker, --mode standalone starts both in one process. Same code, different main() branch, and everything the previous parts built separately (API Server, BoltDB, scheduler, three controllers, node agent) finally meets in one entry point.

The interesting decision is in standalone: the agent talks to the API Server over HTTP even though they share a process. A direct Go call would be faster, but it would create a second code path that nobody tests, and the bugs would live exactly there. Going through localhost makes standalone a special case of multi-node instead of a separate product.
That choice is also why the agent needs registration retry. In the same process the API Server may simply not be listening yet, which is an honest reflection of the fact that no distributed system guarantees startup order.
Where it hurts: standalone is a single point of failure, and it shares a machine, so a container that eats all the memory takes the control plane with it. The retry is capped at 10 attempts, roughly 27.5 seconds, and then gives up silently. actualAddr[0] indexes a string without a length check, so --addr "" panics. And the switch on --mode has no default, so --mode serevr starts, prints nothing and exits 0, which looks exactly like a successful run.
Side notes: Kubernetes ships the same codebase as four separate binaries, k3s crams it into one with server/agent subcommands, and k0s packs containerd and runc inside a single static file. Docker went the other way, splitting one daemon into dockerd/containerd/runc so a daemon restart would not kill containers. The Kubernetes control plane bootstraps itself through static pods, where the kubelet starts the API Server it then registers with. Loopback HTTP is cheaper than it sounds because lo has a 65536-byte MTU and never touches a driver. The 500ms/1s/1.5s backoff here is linear, not exponential, and it is the exact shape that produces a thundering herd once there are fifty agents. And the whole "one binary, many personalities" idea is as old as busybox picking its behavior from argv[0].

Part twenty-nine of the Sheep & Shepherd series.

Graceful Shutdown: Signals, Channels, Cleanup ¶

August 22, 2026

Shepherd runs half a dozen goroutines (scheduler, three controllers, API server, agent), and when SIGTERM lands they all have to stop cleanly. The whole trick is one close(stopCh).

A closed channel hands out the zero value an unlimited number of times, so every goroutine sitting in its own select wakes up at once. That is why close beats a send, which wakes exactly one listener, and why the idiom is a zero-byte chan struct{}: the value carries no meaning, only the event does.
Why a plain channel is cleaner than context.WithCancel for a simple "everything stops" signal, and how <-sigCh blocks main until the signal arrives in standalone mode.
Three holes left open: nothing waits for reconcile to finish, so a controller mid-write to BoltDB can be cut off; close(stopCh) twice panics, so exactly one owner may ever close it; and api.Shutdown(context.Background()) with no deadline hangs forever on a lingering keep-alive.
Side notes: SIGKILL and SIGSTOP are the two signals you cannot trap. Kubernetes' terminationGracePeriodSeconds of 30s and docker stop at 10s follow the same polite SIGTERM-wait-SIGKILL script our close mimics without a timer. PID 1 ignores unhandled SIGTERM entirely, which is the whole reason tini exists. net/http has shipped Server.Shutdown since Go 1.8, and signal.NotifyContext in Go 1.16 folds the signal handler into a context so the manual sigCh disappears.

Part twenty-eight of the Sheep & Shepherd series.

Async Scheduling: Why a Pod Is Created as Pending ¶

August 15, 2026

sheepctl apply returns 201 Created in about two milliseconds, and the pod it just created is not running. It is not even scheduled. It is Pending, always, with no exceptions.

That is not a bug, it is the failure model. The API Server writes the object, fires go scheduler.SchedulePod() and forgets about it, and a reconcile loop ticking every two seconds picks up anything that goroutine dropped.
The synchronous alternative blocks an HTTP worker for seconds, times out when there are no nodes, and lies on top of that: a 200 promising Running is void the moment the agent crashes mid-launch.
pod.Spec.NodeName is the only thing separating "the scheduler has not looked yet" from "the agent has not started yet". The full timeline from apply to Running takes about 3.5 seconds, of which 3 are just waiting for the agent's tick.
Known holes: the fast-path goroutine and the reconcile loop can grab the same pod at once and schedule it twice, because there is no check-and-claim on NodeName. A pod that fits nowhere hangs Pending forever and silently, since no reason is ever written to Status. And with no Watch API there is no wait --for=Ready, only sleep and polling.
Side notes: the Kubernetes scheduler does not PATCH the pod, it POSTs to a separate /binding subresource so RBAC can grant placement without granting spec writes. Its assume step marks a pod as placed in a local cache before the bind returns, otherwise it would sell the same node memory ten times over while one request is in flight. Pending is a family of causes whose real reason lives in the PodScheduled=False condition, not the phase. percentageOfNodesToScore trades placement quality for latency because scoring 5000 nodes is too expensive. And SIG Scalability's SLO promises 99% of pods start within 5 seconds, with image pull deliberately excluded.

Part twenty-seven of the Sheep & Shepherd series.

Label Selectors: Linking Resources Without Foreign Keys ¶

August 8, 2026

In SQL you link tables through foreign keys. In Kubernetes the link between resources runs through labels. A Deployment has no "my pods" field, it has a selector: all pods carrying app=web.

matchLabels is 8 lines of Go implementing AND logic over map[string]string, and those 8 lines connect Deployments to Pods, Services to Pods, and Pods to Nodes through nodeSelector.
Why Kubernetes rejected foreign keys on purpose: a pod is ephemeral, born and killed dozens of times a day with a new UID each time. Any reference by ID goes stale instantly, while labels survive recreation.
Three kinds of label-based link (Deployment to Pods, Service to Pods, Pod to Node), how mergeLabels copies template and selector labels onto new pods, and a comparison table of foreign keys against labels for CRUD operations.
What it costs here: label matching is a full scan on every reconcile, which informers and caches solve in real Kubernetes. There is no label validation. And overlapping selectors let two Deployments fight over the same pods.

Part twenty-six of the Sheep & Shepherd series.

Desired State vs Actual State: The Kubernetes Paradigm ¶

August 1, 2026

The whole Kubernetes architecture rests on one idea: you describe what you want, and the system keeps correcting reality until it matches. Imperative says "create A, then B, then C" and has nothing to say when B crashes. Declarative says "there should be three", and the next loop notices there are two.

Three controllers in Shepherd (replication, service, node) turn out to be the same observe-compare-act body with different nouns.
What that model actually costs: eventual consistency means a pod is never Running instantly; a reconcile that compares with == flaps forever against server-added defaults; there is no backoff, so a failing act spins at full speed burning CPU and logs.
And "self-healing" heals nothing. The deleted pod is not restored. A different pod with a new name, a new IP and an empty disk takes its place.
The idea predates Kubernetes by two decades: CFEngine in 1993, convergent operators, Promise Theory, then Puppet and Terraform.
Side notes: metadata.generation against status.observedGeneration is the built-in "has the controller even seen my change?" that kubectl rollout status runs on. spec and status are physically separate endpoints with separate RBAC, so a controller cannot write spec. Real controllers do not poll at all, using informers plus watch with a resync measured in hours purely as a lost-event safety net. The most common GitOps incident is an HPA and Argo CD taking turns rewriting spec.replicas forever. And kubectl apply once stored your entire previous YAML as a JSON string in an annotation to make three-way merges work.

Part twenty-five of the Sheep & Shepherd series, and the first of the Distributed Systems Patterns block.

Push to Your Own Registry: Layer Creation and Manifest Upload ¶

July 29, 2026

Pull is downloading. Push is building an OCI image out of a local rootfs and shipping it.

Four steps: tar+gzip the rootfs into a layer while hashing the compressed bytes on the fly, upload it as a blob, upload the config JSON as another blob, then PUT the manifest that ties them together.
The digest lands on the compressed bytes, but the config needs the uncompressed hash, the diffID. Two hashes for one layer, and they never match.
Everything that goes wrong here: one layer for the whole rootfs, so the HEAD-check dedup never fires; empty DiffIDs that stricter validators like Harbor or cosign reject; a missing trailing slash on /blobs/uploads/ that some registries answer with 404; an ignored 202 Accepted that means "you are only getting started" and surfaces three steps later as MANIFEST_BLOB_UNKNOWN; an *os.File body that silently becomes Transfer-Encoding: chunked because Go only infers Content-Length for byte readers; and tar mtimes plus gzip levels making the same rootfs hash differently twice.
Side notes: cross-repository blob mount is a real endpoint (?mount=&from=) and it is what Mounted from library/nginx means. The manifest digest covers exact bytes, so pretty-printing breaks pull-by-digest. sha256:44136fa3... is just the two characters {}, and OCI 1.1 named it so SBOMs and Helm charts can fill the mandatory config slot. Re-pushing a tag orphans the old manifest rather than overwriting it. The spec caps manifests at 4 MiB while overlay2 caps layers at 128. zstd layers exist because pull is decompression-bound. And cosign faked attachments through sha256-<hex>.sig tag names until the Referrers API arrived.

Ends with a full image pushed by hand using nothing but curl, tar and shasum, and docker pull consuming it. Part twenty-four of the Sheep & Shepherd series.

Pull from registry: Auth, Manifest Lists, Multi-Arch ¶

July 25, 2026

sheep pull nginx is one command hiding four steps: parse the reference, fetch a bearer token, resolve a manifest list down to one platform, then stream and extract layers. Why nginx is really library/nginx, and how the anonymous token flow splits auth.docker.io from registry-1.docker.io.

Where this implementation breaks: layers download sequentially; the token can expire mid-pull and nothing re-fetches it; linux/amd64 is hardcoded, which turns into a silent exec format error on Apple Silicon or Graviton; the platform match ignores variant; the Manifests[0] fallback can land on an attestation manifest that is not runnable at all; and the auth server URL is hardcoded instead of discovered from WWW-Authenticate, which is why private GHCR or ECR will not work.
Side notes: decode the anonymous token and its own quota is inside it, pull_limit: 100 per 6 hours, signed by the registry. Docker Hub serves blobs through a 307 to a signed CDN URL, and it only works because Go drops Authorization across hosts. Half of nginx:latest's index entries are BuildKit SBOM and provenance attestations with platform: unknown/unknown. Windows images carry foreign layers the registry never stores. Deleting a file in a layer actually means creating a .wh. whiteout. And schema 1's JWS-signed manifests were dropped in favour of plain content-addressing.

Part twenty-three of the Sheep & Shepherd series.

Content-Addressable Storage: SHA256 as the Key ¶

July 22, 2026

In Meadow a blob's file name is the SHA256 of its content. That buys deduplication (identical layers stored once) and verification (hash mismatch means corruption) for free.

io.MultiWriter(tmpFile, h) streams to disk and hashes in a single pass, which forces the temp-file then atomic os.Rename dance: you cannot know the final name until you know the hash.
Manifests are stored twice, by tag and by digest, with a .content-type sidecar because the filesystem will not hold a MIME type.
The downsides are real: no garbage collection, so deleted manifests leave orphaned blobs; os.Rename is only atomic within one filesystem; and content-addressing depends on a strong hash, which is why OCI pins SHA256 after SHA-1 fell.
Side notes: Git has done this since 2005, CAM hardware named it, Docker only adopted content-addressable image IDs in 1.10, and IPFS, Nix and ZFS scale the same idea to networks, packages and blocks.

Part twenty-two of the Sheep & Shepherd series.

OCI Distribution Spec: Writing Our Own Docker Registry ¶

July 18, 2026

A Docker Registry is just an HTTP server with a specific set of endpoints. Meadow implements the OCI Distribution Spec in 370 lines behind a single /v2/ handler that routes by path.

Blob upload with post-write digest verification, where a mismatch means delete and error. Manifest push and pull by tag and by digest.
The enumerated OCI error format (BLOB_UNKNOWN, MANIFEST_UNKNOWN, DIGEST_INVALID) that clients parse by code rather than message, and why LastIndex routing survives repo names with slashes.
Where it breaks: monolithic-only upload holds the whole blob in memory, there is no chunked PATCH, and no pagination.
Side notes: /v2/ is the v1-to-v2 handshake that outlived Registry v1; the spec was pulled out of Docker's HTTP API v2; 202 Accepted means "keep going"; the transport-versus-image-spec split lets the same API ship Helm charts and SBOMs through OCI artifacts and ORAS; cross-repository blob mount exists; and production pins by digest, not tag.

Part twenty-one of the Sheep & Shepherd series.

Event System: Audit Trail for the Cluster ¶

July 15, 2026

When something happens in the cluster, you need to know what, when and why. Shepherd's Event system records Normal and Warning events from the API Server, Scheduler, NodeController and ReplicationController, keyed by nanosecond timestamp so BoltDB orders them chronologically for free.

Events are tied to objects (pod/web-0), not to processes, which is exactly what separates them from stdout logs.
Against real Kubernetes: a 1-hour TTL and aggregation through count, the separate Audit Logging mechanism for security, involvedObject and kubectl describe, the migration to events.k8s.io/v1, and why Events are notifications rather than Event Sourcing.

The final part of the Orchestrator series. Part twenty of the Sheep & Shepherd series.

Pod Lifecycle: From Pending to Running ¶

July 11, 2026

The full asynchronous journey of a pod from sheepctl apply to Running, through the API Server, Scheduler, Agent and runtime, where no component ever waits for another.

The same Pending means "waiting for the Scheduler" without a NodeName and "waiting for the Agent" with one.
Eventual consistency, the same model git uses, turns the 5-15 second latency into a predictable sum of independent tickers rather than a bug.
The one-way Running → Failed transition, and the health-check loop that flips a pod to Failed when a container stops.
Against real Kubernetes: phase as a deliberately coarse abstraction versus the real detail in conditions, the Pulling phase where pods hang on slow registries, initContainers, RestartPolicy and CrashLoopBackOff, terminationGracePeriodSeconds, and why readiness and liveness probes are the thing missing here.

Part nineteen of the Sheep & Shepherd series.

Node Agent: kubelet ¶

July 8, 2026

The Agent is Shepherd's kubelet: registration with retry, heartbeats every 10 seconds, pod reconciliation every 3 seconds, and starting containers for assigned pods.

Structure, resource detection from /proc/meminfo, and the reconcile loop that brings Pending pods to Running.
Why Shepherd's per-container IP is a simplification compared to the shared network namespace Kubernetes uses.
Against real Kubernetes: CRI instead of direct runtime calls, the pause infra container, syncLoop and PLEG, static pods, TLS bootstrap, and why the 110 max-pods limit is really about control-plane load.

Part eighteen of the Sheep & Shepherd series.

Node Health: Heartbeat and Failure Detection ¶

July 4, 2026

How does the control plane know a node is alive? Through heartbeats. The agent sends one every 10 seconds, and if 30 seconds pass in silence the NodeController flips the node to NotReady and the scheduler stops placing pods on it.

Two sides to the mechanism: the agent writing "I am alive", and a separate controller deciding "you are dead".
A heartbeat is not a liveness check, it is an agreement about a timeout. Hence the split-brain problem when only the heartbeat is lost, plus the clock-skew and last-write-wins traps.
Against real Kubernetes: the Lease object that replaced full status rewrites, taint-based eviction with rate limiting, and how the Φ Accrual detector and SWIM gossip go beyond a single hardcoded timeout.

Modeled on Kubernetes node heartbeats. Part seventeen of the Sheep & Shepherd series.

Service Discovery: How a Service Finds Its Pods ¶

July 1, 2026

Pods come and go and their IPs change on every restart. A Service is the stable entry point that always knows where the right pods are.

The ServiceController rebuilds the endpoints list every 5 seconds through label selectors. A pod has to match labels, be Running and have an IP to make the cut.
Which means a Service is really a database record someone recomputes, not a proxy.
Against real Kubernetes: ClusterIP as a virtual IP that exists on no interface, kube-proxy's iptables and IPVS modes, EndpointSlice, headless services, and the eBPF alternative in Cilium.

Modeled on Kubernetes Services. Part sixteen of the Sheep & Shepherd series.

Replication Controller: Scale Up and Scale Down ¶

June 27, 2026

A Deployment says "I want 3 replicas". This is how the ReplicationController makes it so.

Creating pods from a deployment template, the loose app=web label-selector link instead of an owned list, scale up and scale down, updating ReadyReplicas, and the full async path from sheepctl to a running container.
Where index naming (web-0) and overlapping selectors bite, plus where kubectl scale, the HorizontalPodAutoscaler and scale-to-zero fit in.

Modeled on Kubernetes ReplicaSet. Part fifteen of the Sheep & Shepherd series.

Reconciliation Loop: The Heart of Shepherd ¶

June 24, 2026

The reconciliation loop is the heart of any orchestrator: describe the desired state, and controllers keep comparing it with reality and fixing the difference.

The observe-compare-act pattern and why it makes the system self-healing.
Three controllers running in parallel in Shepherd, idempotency as a survival requirement, and where level-triggered beats edge-triggered.

Modeled on Kubernetes controllers. Part fourteen of the Sheep & Shepherd series.

Scheduler: How to Pick a Node for a Pod ¶

June 20, 2026 - A freshly created pod is Pending with no node assigned. How Shepherd's scheduler picks the best node in two phases - filter (drop infeasible nodes) and score (prefer the least-loaded). Resource checks, label matching, least-loaded scoring, and why a pod stays Pending even after it's scheduled. Modeled on the Kubernetes scheduler. Part thirteen of the Sheep & Shepherd series.

BoltDB Instead of etcd: Embedded State Store ¶

June 17, 2026

Kubernetes runs on etcd, but for a learning project that is overkill. Shepherd keeps all cluster state in BoltDB, an embedded key-value store living in a single file.

Buckets as tables, namespaced keys, read and write transactions, watch channels for change notifications, and an event log.
Why it is perfect for learning, and where it falls short of etcd.

Part twelve of the Sheep & Shepherd series.

Kubernetes API Server in 300 Lines ¶

June 13, 2026 - The Kubernetes API Server is the center of the entire cluster. How Shepherd implements REST API with CRUD for pods, services, deployments and nodes in ~300 lines of Go using only net/http. Asynchronous scheduling, namespaced resources, and logging middleware. Part eleven of the Sheep & Shepherd series.

A Docker CLI in 500 Lines of Go ¶

June 10, 2026 - Subcommand routing, flag parsing, and formatted output - all without CLI frameworks. How Sheep implements run, ps, stop, rm and 11 more Docker-like commands in a single file using only the Go standard library. Part ten of the Sheep & Shepherd series.

Container Lifecycle: State Machine from Created to Removed ¶

June 7, 2026 - A container moves through three states - created, running, stopped. How the state machine drives Create/Start/Stop/Remove, why every transition is persisted to state.json, and how signal 0 checks if a container survived a daemon restart. Part nine of the Sheep & Shepherd series.

Image Management: tar Archive to rootfs to Container ¶

June 2, 2026 - A container image is just an archive with a filesystem. How a tar archive becomes a container rootfs through import and bootstrap, how OCI whiteout files delete files across layers, and why Sheep keeps a full rootfs instead of layers. Part eight of the Sheep & Shepherd series.

NAT and iptables: How a Container Sees the Internet ¶

May 30, 2026 - A container's 10.20.0.x address is private - no router will route it. How ip_forward and a single MASQUERADE rule let packets reach the internet and find their way back via conntrack. Part seven of the Sheep & Shepherd series.

Bridge Networking: Giving a Container an IP Address ¶

May 25, 2026 - A container in a fresh network namespace has no network at all, not even loopback. How a Linux bridge, veth pairs, and a touch of NAT give it an IP and an internet route. Part six of the Sheep & Shepherd series.

OverlayFS: Copy-on-Write Layers Like Docker ¶

May 20, 2026 - How OverlayFS stacks a read-only image layer and a per-container read-write layer into one filesystem, and why copy-up lets 10 nginx containers share 100MB instead of each carrying their own. Part five of the Sheep & Shepherd series.