Blog¶

Welcome to my blog! Here you'll find articles about DevOps, AWS, cloud architecture, and more.

Recent Posts¶

Label Selectors: Linking Resources Without Foreign Keys ¶

August 8, 2026 - In SQL you link tables through foreign keys. In Kubernetes the link between resources is through labels. A Deployment doesn't have a "my pods" field — it has a selector: "all pods with labels app=web." matchLabels is 8 lines of Go that implement AND logic over map[string]string, and those 8 lines connect Deployments to Pods, Services to Pods, and Pods to Nodes via nodeSelector. Why Kubernetes deliberately rejected foreign keys: a pod is ephemeral, born and killed dozens of times a day with a new UID each time — any reference by ID goes stale instantly, but labels survive recreation. Three types of label-based links (Deployment→Pods, Service→Pods, Pod→Node), how mergeLabels copies template and selector labels onto new pods, and the comparison table of foreign keys vs labels for CRUD operations. Honest about the gaps: label matching is a full scan on every reconcile (informers and caches solve this in real Kubernetes), there's no label validation, and overlapping selectors let two Deployments fight over the same pods. Part twenty-six of the Sheep & Shepherd series.

Desired State vs Actual State: The Kubernetes Paradigm ¶

August 1, 2026 - The whole Kubernetes architecture rests on one idea: you describe what you want, and the system keeps correcting reality until it matches. Imperative says "create A, then B, then C" and has nothing to say when B crashes; declarative says "there should be three" and the next loop notices there are two. Three controllers in Shepherd — replication, service, node — turn out to be the same observe-compare-act body with different nouns. Honest about the gaps: eventual consistency means a pod is never Running instantly, a reconcile that compares with == flaps forever against server-added defaults, there's no backoff so a failing act spins at full speed burning CPU and logs, and "self-healing" heals nothing — the deleted pod isn't restored, a different pod with a new name, new IP, and empty disk takes its place. Wider context: the idea predates Kubernetes by two decades (CFEngine 1993, convergent operators, Promise Theory, then Puppet and Terraform), metadata.generation vs status.observedGeneration is the built-in "has the controller even seen my change?" that kubectl rollout status runs on, spec and status are physically separate endpoints with separate RBAC so a controller cannot write spec, real controllers don't poll at all (informers plus watch, with a resync measured in hours purely as a lost-event safety net), the most common GitOps incident is an HPA and Argo CD taking turns rewriting spec.replicas forever, and kubectl apply once stored your entire previous YAML as a JSON string in an annotation to make three-way merges work. Part twenty-five of the Sheep & Shepherd series, and the first of the Distributed Systems Patterns block.

Push to Your Own Registry: Layer Creation and Manifest Upload ¶

July 29, 2026 - Pull is downloading; push is building an OCI image out of a local rootfs and shipping it. Four steps: tar+gzip the rootfs into a layer while hashing the compressed bytes on the fly, upload it as a blob, upload the config JSON as another blob, then PUT the manifest that ties them together. Why the digest lands on the compressed bytes but the config needs the uncompressed hash (diffID) — two hashes for one layer, and they never match. Honest about the gaps: one layer for the whole rootfs (so the HEAD-check dedup never fires), empty DiffIDs that stricter validators like Harbor or cosign reject, a missing trailing slash on /blobs/uploads/ that some registries answer with 404, an ignored 202 Accepted that means "you're only getting started" and surfaces three steps later as MANIFEST_BLOB_UNKNOWN, an *os.File body that silently becomes Transfer-Encoding: chunked because Go only infers Content-Length for byte readers, and tar mtimes plus gzip levels making the same rootfs hash differently twice. Wider context: cross-repository blob mount is a real endpoint (?mount=&from=) and it's what Mounted from library/nginx means, the manifest digest is over exact bytes so pretty-printing breaks pull-by-digest, sha256:44136fa3… is just the two characters {} and OCI 1.1 named it so SBOMs and Helm charts can fill the mandatory config slot, re-pushing a tag orphans the old manifest rather than overwriting it, the spec caps manifests at 4 MiB while overlay2 caps layers at 128, zstd layers exist because pull is decompression-bound, and cosign faked attachments through sha256-<hex>.sig tag names until the Referrers API arrived. Ends with a full image pushed by hand using nothing but curl, tar, and shasum — and docker pull consuming it. Part twenty-four of the Sheep & Shepherd series.

Pull from registry: Auth, Manifest Lists, Multi-Arch ¶

July 25, 2026 - sheep pull nginx is one command hiding four steps: parse the reference, fetch a bearer token, resolve a manifest list down to one platform, then stream and extract layers. Why nginx is really library/nginx, and how the anonymous token flow splits auth.docker.io from registry-1.docker.io. Honest about the gaps: layers download sequentially, the token can expire mid-pull and nothing re-fetches it, linux/amd64 is hardcoded (silent exec format error on Apple Silicon or Graviton), the platform match ignores variant, the Manifests[0] fallback can land on an attestation manifest that isn't runnable at all, and the auth server URL is hardcoded instead of discovered from WWW-Authenticate — which is why private GHCR or ECR won't work. Wider context: decode the anonymous token and its own quota is inside it (pull_limit: 100 per 6 hours, signed by the registry), Docker Hub serves blobs via a 307 to a signed CDN URL (and it only works because Go drops Authorization across hosts), half of nginx:latest's index entries are BuildKit SBOM/provenance attestations with platform: unknown/unknown, Windows images carry foreign layers the registry never stores, deleting a file in a layer actually means creating a .wh. whiteout, and schema 1's JWS-signed manifests were dropped in favour of plain content-addressing. Part twenty-three of the Sheep & Shepherd series.

Content-Addressable Storage: SHA256 as the Key ¶

July 22, 2026 - In Meadow, a blob's file name is the SHA256 of its content — which buys deduplication (identical layers stored once) and verification (hash mismatch = corruption) for free. How io.MultiWriter(tmpFile, h) streams to disk and hashes in a single pass, why that forces the temp-file → atomic os.Rename dance (you can't know the final name until you know the hash), and how manifests are stored twice — by tag and by digest — with a .content-type sidecar because the filesystem won't hold a MIME type. Honest downsides: no garbage collection (deleted manifests leave orphaned blobs), os.Rename only atomic within one filesystem, and content-addressing's dependence on a strong hash (SHA-1 fell, which is why OCI pins SHA256). Wider context: Git has done this since 2005, CAM hardware named it, Docker only adopted content-addressable image IDs in 1.10, and IPFS/Nix/ZFS scale the same idea to networks, packages, and blocks. Part twenty-two of the Sheep & Shepherd series.

OCI Distribution Spec: Writing Our Own Docker Registry ¶

July 18, 2026 - A Docker Registry is just an HTTP server with a specific set of endpoints — Meadow implements the OCI Distribution Spec in 370 lines behind a single /v2/ handler that routes by path. Blob upload with post-write digest verification (mismatch → delete and error), manifest push/pull by tag and digest, the enumerated OCI error format (BLOB_UNKNOWN, MANIFEST_UNKNOWN, DIGEST_INVALID) that clients parse by code not message, and why LastIndex routing survives repo names with slashes. Where it breaks: monolithic-only upload holds the whole blob in memory, no chunked PATCH, no pagination. Comparison with the wider ecosystem: /v2/ as the v1→v2 handshake that outlived Registry v1, the spec pulled out of Docker's HTTP API v2, 202 Accepted meaning "keep going", the transport-vs-image-spec split that lets the same API ship Helm charts and SBOMs (OCI artifacts / ORAS), cross-repository blob mount, and why production pins by digest, not tag. Part twenty-one of the Sheep & Shepherd series.

Event System: Audit Trail for the Cluster ¶

July 15, 2026 - When something happens in the cluster, you need to know what, when, and why. How Shepherd's Event system records Normal and Warning events from the API Server, Scheduler, NodeController, and ReplicationController — keyed by nanosecond timestamp for free chronological ordering in BoltDB. Why events are tied to objects (pod/web-0), not processes, and how that differs from stdout logs. Comparison with real Kubernetes: 1-hour TTL and aggregation with count, the separate Audit Logging mechanism for security, involvedObject and kubectl describe, migration to events.k8s.io/v1, and why Events are notifications — not Event Sourcing. The final part of the Orchestrator series. Part twenty of the Sheep & Shepherd series.

Pod Lifecycle: From Pending to Running ¶

July 11, 2026 - The full asynchronous journey of a pod from sheepctl apply to Running — through the API Server, Scheduler, Agent, and runtime, where no component ever waits for another. Why the same Pending means "waiting for the Scheduler" without a NodeName and "waiting for the Agent" with one, how eventual consistency (the same model as git) makes the 5-15 second latency a predictable sum of independent tickers rather than a bug, the one-way Running → Failed transition, and the health-check loop that flips a pod to Failed when a container stops. Comparison with real Kubernetes: phase as a deliberately coarse abstraction versus the real detail in conditions, the Pulling phase where pods hang on slow registries, initContainers, RestartPolicy and CrashLoopBackOff, terminationGracePeriodSeconds, and why readiness/liveness probes are what's missing. Part nineteen of the Sheep & Shepherd series.

Node Agent: kubelet ¶

July 8, 2026 - The Agent is Shepherd's kubelet: registration with retry, heartbeats every 10 seconds, pod reconciliation every 3 seconds, and starting containers for assigned pods. Structure, resource detection from /proc/meminfo, the reconcile loop that brings Pending pods to Running, and why Shepherd's per-container IP is a simplification compared to Kubernetes's shared network namespace. Comparison with real Kubernetes: CRI instead of direct runtime calls, the pause infra container, syncLoop and PLEG, static pods, TLS bootstrap, and why 110 max-pods is about control-plane load. Part eighteen of the Sheep & Shepherd series.

Node Health: Heartbeat and Failure Detection ¶

July 4, 2026 - How does the control plane know a node is alive? Through heartbeats — the agent sends one every 10 seconds, and if 30 seconds pass in silence the NodeController flips the node to NotReady and the scheduler stops placing pods on it. The two sides of the mechanism (the agent that writes "I'm alive" and the separate controller that decides "you're dead"), why a heartbeat isn't a liveness check but an agreement about a timeout, the split-brain problem when only the heartbeat is lost, and the clock-skew and last-write-wins traps. Comparison with real Kubernetes: the Lease object that replaced full status rewrites, taint-based eviction with rate limiting, and how the Φ Accrual detector and SWIM gossip go beyond a single hardcoded timeout. Modeled on Kubernetes node heartbeats. Part seventeen of the Sheep & Shepherd series.

Service Discovery: How a Service Finds Its Pods ¶

July 1, 2026 - Pods come and go and their IPs change on every restart — a Service is the stable entry point that always knows where the right pods are. How the ServiceController rebuilds the endpoints list every 5 seconds through label selectors, why a pod needs to match labels, be Running, and have an IP to make the cut, and why a Service is really just a database record someone recomputes — not a proxy. Comparison with real Kubernetes: ClusterIP as a virtual IP that exists on no interface, kube-proxy's iptables/IPVS modes, EndpointSlice, headless services, and the eBPF (Cilium) alternative. Modeled on Kubernetes Services. Part sixteen of the Sheep & Shepherd series.

Replication Controller: Scale Up and Scale Down ¶

June 27, 2026 - A Deployment says "I want 3 replicas" — how the ReplicationController makes it so. Creating pods from a deployment template, the loose app=web label-selector link instead of an owned list, scale up and scale down, updating ReadyReplicas, and the full async path from sheepctl to a running container. Why index naming (web-0) and overlapping selectors bite, plus where kubectl scale, the HorizontalPodAutoscaler, and scale-to-zero fit in. Modeled on Kubernetes ReplicaSet. Part fifteen of the Sheep & Shepherd series.

Reconciliation Loop: The Heart of Shepherd ¶

June 24, 2026 - The reconciliation loop is the heart of any orchestrator: describe the desired state, and controllers constantly compare it with reality and fix the difference. The observe-compare-act pattern, why it makes the system self-healing, the three controllers running in parallel in Shepherd, idempotency as a survival requirement, and where level-triggered beats edge-triggered. Modeled on Kubernetes controllers. Part fourteen of the Sheep & Shepherd series.

Scheduler: How to Pick a Node for a Pod ¶

June 20, 2026 - A freshly created pod is Pending with no node assigned. How Shepherd's scheduler picks the best node in two phases — filter (drop infeasible nodes) and score (prefer the least-loaded). Resource checks, label matching, least-loaded scoring, and why a pod stays Pending even after it's scheduled. Modeled on the Kubernetes scheduler. Part thirteen of the Sheep & Shepherd series.

BoltDB Instead of etcd: Embedded State Store ¶

June 17, 2026 - Kubernetes runs on etcd, but for a learning project that's overkill. How Shepherd stores all cluster state in BoltDB — an embedded key-value store in a single file. Buckets as tables, namespaced keys, read/write transactions, watch channels for change notifications, and an event log. Why it's perfect for learning and where it falls short of etcd. Part twelve of the Sheep & Shepherd series.

Kubernetes API Server in 300 Lines ¶

June 13, 2026 - The Kubernetes API Server is the center of the entire cluster. How Shepherd implements REST API with CRUD for pods, services, deployments and nodes in ~300 lines of Go using only net/http. Asynchronous scheduling, namespaced resources, and logging middleware. Part eleven of the Sheep & Shepherd series.

A Docker CLI in 500 Lines of Go ¶

June 10, 2026 - Subcommand routing, flag parsing, and formatted output — all without CLI frameworks. How Sheep implements run, ps, stop, rm and 11 more Docker-like commands in a single file using only the Go standard library. Part ten of the Sheep & Shepherd series.

Container Lifecycle: State Machine from Created to Removed ¶

June 7, 2026 - A container moves through three states — created, running, stopped. How the state machine drives Create/Start/Stop/Remove, why every transition is persisted to state.json, and how signal 0 checks if a container survived a daemon restart. Part nine of the Sheep & Shepherd series.

Image Management: tar Archive to rootfs to Container ¶

June 2, 2026 - A container image is just an archive with a filesystem. How a tar archive becomes a container rootfs through import and bootstrap, how OCI whiteout files delete files across layers, and why Sheep keeps a full rootfs instead of layers. Part eight of the Sheep & Shepherd series.

NAT and iptables: How a Container Sees the Internet ¶

May 30, 2026 - A container's 10.20.0.x address is private — no router will route it. How ip_forward and a single MASQUERADE rule let packets reach the internet and find their way back via conntrack. Part seven of the Sheep & Shepherd series.

Bridge Networking: Giving a Container an IP Address ¶

May 25, 2026 - A container in a fresh network namespace has no network at all — not even loopback. How a Linux bridge, veth pairs, and a touch of NAT give it an IP and an internet route. Part six of the Sheep & Shepherd series.

OverlayFS: Copy-on-Write Layers Like Docker ¶

May 20, 2026 - How OverlayFS stacks a read-only image layer and a per-container read-write layer into one filesystem — and why copy-up lets 10 nginx containers share 100MB instead of each carrying their own. Part five of the Sheep & Shepherd series.

Cgroups v2: Limiting Memory, CPU, and PIDs ¶

May 15, 2026 - Namespaces isolate but don't limit. How memory.max, cpu.max, and pids.max cap container resources through the cgroups v2 virtual filesystem — part four of the Sheep & Shepherd series.

pivot_root: How a Container Gets Its Own Filesystem ¶

May 9, 2026 - How pivot_root(2) swaps a process's root directory at the mount namespace level — and why it's the proper isolation primitive instead of chroot. Part three of the Sheep & Shepherd series.

Re-Exec Pattern: Why Go and clone() Don't Get Along ¶

May 2, 2026 - Go's threading model conflicts with clone(). The self re-exec pattern fixes it — part two of the Sheep & Shepherd series.

Linux Namespaces: Isolating a Process in 50 Lines of Go ¶

April 28, 2026 - A container is a process with a restricted view of the system. How to isolate a process using Linux namespaces in 50 lines of Go — the first part of the Sheep & Shepherd series.

AI sovereignty: your own model on DGX Spark instead of an API ¶

April 18, 2026 - How I stopped paying OpenAI and moved inference onto my own DGX Spark box with vLLM. About the hardware, the CUDA/PyTorch pain, an honest comparison with Ollama, and a small web UI to run it all.

EMM: LangGraph traces in Phoenix ¶

April 11, 2026 - One init at startup covers 15 LangGraph agents. Manual spans extend coverage to voice (Gemini Live tools), avatar (Runway sessions), and Izabella chat (OpenAI/Ollama/Google + MCP tool loop).

EMM: A2A Inspector in the app and MCP for it ¶

April 8, 2026 - Built-in UI plus an MCP server: inspect the Agent Card, run tasks/submit and tasks/status from the IDE without leaving the monorepo.

EMM A2A Phase 4: Auth, Rate Limiting, Observability ¶

April 4, 2026 - X-API-Key, rate limiting, structured logging. A2A endpoints now protected like other APIs.

AI Reliability Engineering — certification from fwdays ¶

March 31, 2026 - Completed the AI Reliability Engineering course from fwdays. Why AI system reliability belongs in the same conversation as classic SRE.

EMM A2A Phase 3: Stream task status ¶

March 28, 2026 - SSE instead of polling. GET /api/a2a/tasks/{id}/stream. Theory, diagrams, capabilities.streaming.

EMM A2A Phase 2+: TaskStore and tasks/status ¶

March 21, 2026 - A2A task lifecycle: submit → taskId → poll status. InMemoryTaskStore, 1h TTL. Diagrams.

Production Technology Risks: Planning for When Dependencies Fail ¶

March 14, 2026 - PostgreSQL, MinIO, lakeFS: when choosing production technologies, think beyond features — what happens in 5 years?

EMM A2A Phase 2: Task Manager (list_board)¶

March 14, 2026 - Second skill — list_board. Routing by skillId, interaction diagrams, what changed.

EMM A2A Phase 1: Process Manager as A2A Server ¶

March 7, 2026 - Process Manager is the first agent with an A2A interface. Protocol theory, interaction diagrams, what's implemented.

How I Became an AWS Community Builder ¶

March 5, 2026 - A few years ago it was regular DevOps — deploys, scripts. Then I started thinking in clusters instead of servers. Here's how that led to AWS Community Builders.

Developing and Testing AI Agents: From LangGraph to Production ¶

February 12, 2026 - How to write, test, and debug LangGraph agents? Which patterns work for StateGraph? Why are pytest fixtures critical? Development workflow from first code to production deployment.

Data Versioning for AI Agents: Real-World Experience with lakeFS ¶

February 8, 2026 - When AI agents start moving your files around, version control stops being theoretical. Here's how integrating lakeFS changed my approach to data management in a LangGraph-based agent platform.

Kubernetes Deployment for AI Agents: Real-World Experience with LangGraph ¶

February 3, 2026 - When AI agents move from local Docker Compose to Kubernetes, questions emerge about service discovery, caching, secrets management. How I deployed 7 microservices with minimal downtime.

Building MCP server for self-hosted Jira and Confluence.¶

November 19, 2025 - Building MCP server for self-hosted Jira and Confluence.

Izabella. Create agentic tools. Convertor from pdf to fb2 format.¶

October 30, 2025 - Izabella. Create agentic tools. Convertor from pdf to fb2 format.

Building a multi-site apartment searcher: Design patterns and architecture ¶

October 16, 2025 - Building a multi-site apartment searcher: Design patterns and architecture.

Tarot AI Agent: Innovative Approach to Risk Assessment Through Artificial Intelligence ¶

September 11, 2025 - Tarot AI Agent: Innovative Approach to Risk Assessment Through Artificial Intelligence.

Using AWS ECR as a Universal OCI Repository ¶

July 10, 2025 - Using AWS ECR as a universal OCI repository for storing various types of artifacts.

Cert Manager in Kubernetes ¶

June 3, 2025 - Setting up and using Cert Manager for automatic management of SSL certificates in Kubernetes.

New Architecture for Isabella - C4 Diagrams ¶

May 14, 2025 - Development of a new system architecture using C4 diagrams.

New Architecture for Isabella - Structure ¶

May 14, 2025 - Detailed structure analysis for the new Isabella architecture.

New Architecture for Isabella ¶

May 10, 2025 - Overview of the new architecture design for Isabella project.

Waste Resources - Financial Optimization in the Cloud ¶

April 14, 2025 - Analysis and optimization of cloud resource costs.

Redis Backup on AWS S3 ¶

March 19, 2025 - Automation of Redis backup to AWS S3.

RDS Import with Terraform ¶

February 17, 2025 - Importing existing RDS instances into Terraform.

RDS Import with Terraform (EN)¶

February 17, 2025 - Importing existing RDS instances into Terraform (English version).

AWS Lambda Cost Optimization ¶

February 26, 2025 - Strategies for optimizing AWS Lambda costs.

Kubernetes Onboarding with Flux ¶

March 10, 2025 - Automating Kubernetes onboarding using Flux.

RDS Migration Cases ¶

February 9, 2025 - Various scenarios for migrating databases to AWS RDS.

SPA Deployment on S3 with CloudFront ¶

February 12, 2025 - Deploying Single Page Application on AWS S3 with CloudFront.

Karpenter Properties ¶

December 8, 2024 - Properties and configuration of Karpenter for Kubernetes.

AI Stable Diffusion ¶

February 8, 2025 - Using Stable Diffusion for image generation.

ARM vs AMD ¶

December 5, 2024 - Comparison of ARM and AMD architectures for cloud solutions.