Skip to content

Embedded vs External DB: BoltDB vs etcd Trade-offs

Embedded vs External DB: BoltDB vs etcd Trade-offs

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

Kubernetes uses etcd - a distributed key-value store with Raft consensus. Shepherd uses BoltDB - an embedded database in a single file. Both store key-value pairs, but the difference lies in what happens during failures.

We wrote the BoltDB store back in part twelve and have simply been using it ever since: the API Server reads and writes through it, the scheduler and the controllers compare desired against actual state through it, the event log lives in its own bucket. And the previous part made it plain that the entire cluster state lives in one file belonging to exactly one process. Time to say the price of that out loud.

BoltDB: one line, zero dependencies

func NewStore(path string) (*Store, error) {
    db, err := bolt.Open(path, 0600,
        &bolt.Options{Timeout: 1 * time.Second})
    if err != nil {
        return nil, fmt.Errorf("open store: %w", err)
    }

    err = db.Update(func(tx *bolt.Tx) error {
        for _, b := range [][]byte{
            bucketPods, bucketServices,
            bucketDeployments, bucketNodes, bucketEvents,
        } {
            tx.CreateBucketIfNotExists(b)
        }
        return nil
    })

    return &Store{db: db}, nil
}

bolt.Open(), and the database is ready. One file on disk, ~100KB for an empty cluster, a few MB with dozens of resources. No cluster, no service discovery, no TLS between nodes - none of that exists as a problem. We covered buckets and namespaced keys separately; what matters here is only how this differs from etcd.

Comparison

BoltDB etcd
Deployment bolt.Open("file.db") Cluster of 3-5 nodes
Size ~3MB (Go module) ~50MB (binary)
Transactions ACID, serializable Linearizable reads/writes
Readers Concurrent (MVCC) Concurrent
Writers One at a time (mutex) Through Raft consensus
Replication None Automatic (Raft)
Watch Manual (Go channels) Built-in (gRPC stream)
Backup cp file.db backup.db etcdctl snapshot save
Fault tolerance Goes down = everything goes down 1 of 3 goes down = still works

How BoltDB is used in Shepherd

All operations are read or write transactions:

// Write: Update() - exclusive access
func (s *Store) put(bucket []byte, key []byte, v any) error {
    data, _ := json.Marshal(v)
    return s.db.Update(func(tx *bolt.Tx) error {
        return tx.Bucket(bucket).Put(key, data)
    })
}

// Read: View() - concurrent access
func (s *Store) get(bucket []byte, key []byte, v any) error {
    return s.db.View(func(tx *bolt.Tx) error {
        data := tx.Bucket(bucket).Get(key)
        if data == nil {
            return fmt.Errorf("not found")
        }
        return json.Unmarshal(data, v)
    })
}

View() doesn't block other View() calls. Multiple goroutines can read at the same time. But Update() blocks everything - both readers and writers. For Shepherd with 4 controller goroutines, this is acceptable since each write takes milliseconds.

Watch: the biggest difference

In Shepherd, Watch is implemented through Go channels:

type Store struct {
    db             *bolt.DB
    podWatchers    []chan Event
    watchMu        sync.Mutex
}

func (s *Store) WatchPods() chan Event {
    s.watchMu.Lock()
    defer s.watchMu.Unlock()
    ch := make(chan Event, 64)
    s.podWatchers = append(s.podWatchers, ch)
    return ch
}

func (s *Store) notify(watchers []chan Event, evt Event) {
    s.watchMu.Lock()
    defer s.watchMu.Unlock()
    for _, ch := range watchers {
        select {
        case ch <- evt:
        default: // channel full - skip
        }
    }
}

This works within a single process. If the API Server restarts, all watchers are lost and need to subscribe again.

graph LR
    subgraph "BoltDB Watch (Shepherd)"
        W1["Store.UpdatePod()"] --> N["notify()"]
        N --> CH1["chan Event (buffer 64)"]
        N --> CH2["chan Event (buffer 64)"]
        CH1 --> C1["Controller 1"]
        CH2 --> C2["Controller 2"]
    end

    subgraph "etcd Watch (Kubernetes)"
        W2["etcd.Put()"] --> RAFT["Raft log"]
        RAFT --> WATCH["Watch Stream"]
        WATCH --> G1["gRPC stream (client 1)"]
        WATCH --> G2["gRPC stream (client 2)"]
        G1 --> K1["Controller (can be on another machine)"]
        G2 --> K2["Controller (can be on another machine)"]
    end

etcd watch is a gRPC stream with revision tracking. Client disconnected and reconnected? etcd sends all missed changes starting from the last revision. BoltDB has nothing like that - missed = lost. That's why Shepherd controllers use ticker-based polling as the primary mechanism and watch only for faster reaction.

This is exactly why our reconciliation loop is built on periodically comparing the full state rather than on a stream of events. A loop that re-compares desired against actual every time heals itself from a dropped event - it simply sees the divergence on the next tick. A loop that trusts event delivery does not. We went through the same reasoning in async scheduling: if you're eventually consistent anyway, a lost notification is a delay, not state corruption.

When BoltDB is enough

  • Single control plane (single node) - that is, standalone or server+agents with one control plane
  • Hundreds of resources (not tens of thousands)
  • API Server downtime = cluster downtime (acceptable for dev/test)
  • Backup = cp shepherd.db ~/backup/

When you need etcd

  • HA control plane (3+ API Servers)
  • Tens of thousands of pods and hundreds of nodes
  • Leader election between controllers (ours are unique by construction - exactly one of each, in one process)
  • Can't lose state when a single node goes down
  • Need watch with delivery guarantees

What we skipped with both

BoltDB: one writer at a time means that under load, writers queue up. For Shepherd with 4 controllers, that's fine. For Kubernetes with 30 controllers - bottleneck.

etcd: requires 3-5 nodes for quorum. Every write goes through Raft - minimum 2 disk syncs (leader + one follower). For a small cluster, etcd can be more complex than the cluster itself.

K3s from Rancher solved this the same way we did: replaced etcd with SQLite (embedded) for single-node. One file, zero dependencies, full Kubernetes API compatibility. BoltDB for Shepherd is the same approach.

💡 Fun facts

  • BoltDB is a Go port of LMDB (Lightning Memory-Mapped Database). Ben Johnson froze the original boltdb/bolt, so the whole world moved to the etcd-io/bbolt fork, maintained by the etcd team itself. So even the "embedded alternative to etcd" lives under etcd's wing.
  • etcd uses bbolt as its local storage engine. Every etcd node is, internally, that same BoltDB - with Raft, revision-based MVCC, and gRPC watch layered on top.
  • BoltDB is a B+tree with a single writer and copy-on-write pages. That's where ACID comes "for free": a write goes into new pages while the old ones stay valid for concurrent readers until the transaction commits.
  • Raft, which etcd is built on, was created by Diego Ongaro and John Ousterhout specifically as an "understandable alternative to Paxos" - the name comes from "Reliable, Replicated, Redundant, And Fault-Tolerant."
  • Quorum isn't "most nodes", it's "most of the nodes that get to vote", which is why an even cluster size is worse than an odd one. Four nodes tolerate the same single failure three do (quorum 3-of-4 vs 2-of-3) while adding one more thing that can break. That's why the docs always say 3 or 5, never 4.
  • etcd's database size is capped at 2 GB by default (--quota-backend-bytes). Blow through it and the cluster flips into a read-only NOSPACE alarm, at which point Kubernetes abruptly stops accepting any writes. Recovery is compaction + defrag + a manual etcdctl alarm disarm. Our .db file just grows until the disk runs out - worse for control, but with no "cluster is up but accepts nothing" mode.
  • History in etcd isn't infinite: compaction discards old revisions, and a client that tries to start a watch from too old a revision gets ErrCompacted. So even "watch with delivery guarantees" only guarantees delivery within a retention window - the window is just measured in minutes rather than being zero, like ours.
  • MVCC in BoltDB is free, but not free for the disk: pages still visible to a long read transaction can't go back on the freelist. One forgotten View() held open for an hour and the file bloats by every write made during that hour. It's a known class of problem in bbolt, which is where FreelistType and NoFreelistSync came from.
  • A BoltDB file is an mmap. Database size is bounded by the process address space, so on 32-bit platforms the maximum is ~2 GB (bbolt even has a maxMapSize constant for it). On 64-bit it's effectively unbounded.
  • The first two blocks of any .db file are two meta pages carrying a transaction id and a checksum. A commit writes a new meta page over the older of the two; that double buffering is what makes a crash mid-write safe. It's why bolt.Open() on a torn file doesn't panic - it just falls back to the previous valid state.
  • etcdctl snapshot save does under the hood exactly what we recommend instead of cp: it calls Tx.WriteTo() on its own bbolt inside a read transaction. The difference isn't the mechanism but the fact that etcd adds a hash and a revision to the snapshot, so it can verify integrity on restore.
  • K3s went further than SQLite: kine is a shim that speaks the etcd API on top of SQLite, Postgres, or MySQL. Meaning you can run Kubernetes with no etcd at all, just by handing it something that talks its language. Putting our Store behind an interface is the same idea in miniature.
  • Consul and Zookeeper solve the same problem etcd does, and also via a consensus log (Raft and Zab respectively), but with different promises about reads. Zookeeper reads may return slightly stale data by default; getting a guaranteed-fresh one requires an explicit sync. So "distributed KV" isn't one semantic, it's a family of them.

What I figured out while digging into this

The biggest aha moment was that BoltDB watch and etcd watch are "the same thing" only in name. In our case, watch is an in-memory Go channel: restart the process and the subscriptions are gone. In etcd, watch has a revision, so after a reconnect the client catches up on everything it missed.

That's exactly why I made polling the primary mechanism in Shepherd and watch merely an accelerator. At first it felt like a "backward" design. Then it clicked: this is the very eventual-consistency model, just honestly admitted - I'm not pretending to have a delivery guarantee I don't actually have.

What to watch out for

  • BoltDB has exactly one writer. A long write transaction (e.g. iterating over a large bucket inside Update()) blocks all other writes - keep write transactions short.
  • bolt.Open() takes an exclusive file lock. Two processes on the same .db file, and the second hangs on Timeout. This is the classic trap when you start standalone and server on the same --data-dir.
  • Our watch channels have a buffer of 64 and a select/default: under a burst of events, messages are silently dropped. Without polling as a backstop, controllers would miss changes.
  • Backup via cp is only safe on a closed or quiet database: a copy of a "hot" file under active writes can be inconsistent. The right way is Tx.WriteTo() inside a read transaction.
  • A read transaction you forgot to close pins pages against reuse. A long View() (say, while streaming a large API response) bloats the file by exactly the volume written during that time. The rule is simple: a View() lives no longer than it takes to copy the data into memory.
  • The file never shrinks on its own. Delete a thousand pods and the file stays the same size, just with free pages inside. Handing the space back to the OS takes a separate pass (rewriting the database into a new file).

What could be improved

  • Add a revision/sequence to events so watch can serve "everything after revision N" - a step toward etcd's semantics.
  • Move snapshot backup to db.View(func(tx) { tx.WriteTo(w) }) instead of cp, to make consistent copies on a live database.
  • Introduce compaction/retention for the events bucket: right now events grow unbounded and bloat the file.
  • As a next exercise - hide Store behind an interface and add a second implementation on top of etcd. Then you can compare the same code on an embedded vs a distributed backend. That's also the only way to find out how many assumptions about "one writer" and "all in one process" have already leaked into the controllers.
  • Add a periodic background snapshot (that same Tx.WriteTo()) with copy rotation - right now backup exists only as a command in the README.

Try it yourself

# BoltDB: one file, the entire cluster state:
ls -lh /var/lib/shepherd/shepherd.db
# Backup:
cp /var/lib/shepherd/shepherd.db ~/shepherd-backup.db
# Check the contents through the API:
curl -s localhost:9876/api/v1/info | jq .
curl -s localhost:9876/api/v1/pods | jq 'length'
curl -s localhost:9876/api/v1/events | jq '.[0:3]'

Starting the Go Systems Programming series. Next up - build tags for cross-platform support.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Two-Mode Architecture