Skip to content

Push to Your Own Registry: Layer Creation and Manifest Upload

Push to Your Own Registry: Layer Creation and Manifest Upload

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

Pull is downloading. Push is creating an OCI image from a local rootfs and sending it to a registry. Four steps: create the layer, create the config, assemble the manifest, send everything.

graph LR
    A["rootfs/"] -->|"tar.gz"| B["Layer blob"]
    C["Image metadata"] -->|"JSON"| D["Config blob"]
    B & D --> E["OCI Manifest"]
    E --> F["Registry"]

Step 1: Layer from rootfs

func createLayer(rootfs string, w io.Writer) (string, int64, error) {
    h := sha256.New()
    countWriter := &countingWriter{
        w: io.MultiWriter(w, h),
    }

    gw := gzip.NewWriter(countWriter)
    tw := tar.NewWriter(gw)

    filepath.Walk(rootfs, func(path string, info os.FileInfo,
        err error) error {
        rel, _ := filepath.Rel(rootfs, path)
        if rel == "." { return nil }

        header, _ := tar.FileInfoHeader(info, "")
        header.Name = "./" + rel

        if info.Mode()&os.ModeSymlink != 0 {
            link, _ := os.Readlink(path)
            header.Linkname = link
            header.Typeflag = tar.TypeSymlink
        }

        tw.WriteHeader(header)

        if info.Mode().IsRegular() {
            f, _ := os.Open(path)
            io.Copy(tw, f)
            f.Close()
        }
        return nil
    })

    tw.Close()
    gw.Close()

    digest := "sha256:" + hex.EncodeToString(h.Sum(nil))
    return digest, countWriter.n, nil
}

Walk the rootfs, pack into tar, compress with gzip, compute SHA256 on the fly.

Note where the hash is taken: the MultiWriter sits after gzip, so we're hashing the compressed bytes. That's the layer digest — the name the blob gets in the registry. The hash of the uncompressed tar is a different number, called the diffID, and it belongs in the config. We'll come back to why both exist.

Step 2: Upload the layer blob

func uploadBlob(client *http.Client, registryURL, repo,
    filePath, digest string) error {
    // Check if the blob already exists
    headURL := fmt.Sprintf("%s/v2/%s/blobs/%s",
        registryURL, repo, digest)
    headResp, err := client.Head(headURL)
    if err == nil && headResp.StatusCode == http.StatusOK {
        return nil // already exists, skip
    }

    // Upload
    f, _ := os.Open(filePath)
    defer f.Close()

    url := fmt.Sprintf(
        "%s/v2/%s/blobs/uploads?digest=%s",
        registryURL, repo, digest)
    req, _ := http.NewRequest("POST", url, f)
    req.Header.Set("Content-Type", "application/octet-stream")
    client.Do(req)

    return nil
}

HEAD before upload is an optimization. If the layer already exists in the registry (from a previous push), there's no need to upload it again.

Step 3: Config blob

func createImageConfig(img *Image) ociImageConfig {
    return ociImageConfig{
        Created:      img.CreatedAt.Format(time.RFC3339),
        Architecture: "amd64",
        OS:           "linux",
        Config:       map[string]interface{}{},
        RootFS: ociRootFS{
            Type:    "layers",
            DiffIDs: []string{},
        },
    }
}

The config is JSON with image metadata. It's also uploaded as a blob.

Step 4: Manifest

manifest := map[string]any{
    "schemaVersion": 2,
    "mediaType":
        "application/vnd.oci.image.manifest.v1+json",
    "config": map[string]any{
        "mediaType":
            "application/vnd.oci.image.config.v1+json",
        "digest": configDigest,
        "size":   len(configJSON),
    },
    "layers": []map[string]any{
        {
            "mediaType":
                "application/vnd.oci.image.layer.v1.tar+gzip",
            "digest": layerDigest,
            "size":   layerSize,
        },
    },
}

manifestJSON, _ := json.Marshal(manifest)
uploadManifest(client, registryURL, ref.Repo,
    ref.Tag, manifestJSON)

The manifest ties the config and layers together. The registry stores the manifest by tag (latest) and by digest.

Which means those exact manifestJSON bytes matter: the manifest's digest is computed over them, byte for byte. Pretty-print the same structure and you get a valid manifest with a different digest — and docker pull image@sha256:... against the old one breaks. This is why the registry hands back a Docker-Content-Digest header on the PUT instead of expecting you to recompute it.

Full push flow

$ sheep push localhost:5000/myapp:v1
pushing to localhost:5000/myapp:v1
creating layer from rootfs...
uploading layer sha256:a1b2c3d4... (45.2 MB)...
uploading config...
uploading manifest...
pushed localhost:5000/myapp:v1

Where the gotchas are

One layer for the entire rootfs. Docker creates separate layers for each Dockerfile instruction. That gives you deduplication and incremental pushes. Our approach packs the entire rootfs as a single layer every time.

  • The DiffIDs in the config are empty, even though OCI requires the uncompressed layer hashes there. Some registries and runtimes will swallow it, but stricter validation (e.g. cosign or Harbor) will reject such a manifest.
  • Push order is critical: all blobs first (layers + config), the manifest last. If you upload a manifest that references a not-yet-uploaded blob, the registry responds with MANIFEST_BLOB_UNKNOWN and rejects it.
  • The upload URL is missing a trailing slash. The spec route is /v2/<name>/blobs/uploads/ — with the slash. Without it, some registries answer 404 and the push dies for a reason that has nothing to do with your bytes.
  • Single-request monolithic upload (POST .../uploads/?digest=) is the optional form. A registry is allowed to answer 202 Accepted instead of 201 Created, meaning "I opened a session, now PUT the bytes to the Location I gave you." Our code ignores the status code entirely, so on such a registry it reports success while nothing was committed — and the failure surfaces later as MANIFEST_BLOB_UNKNOWN on the manifest, far from the actual cause.
  • Passing an *os.File as the body means Go doesn't know Content-Length. http.NewRequest only infers the length for *bytes.Buffer, *bytes.Reader, and *strings.Reader; anything else gets Transfer-Encoding: chunked. A monolithic upload is specified with an explicit Content-Length, and several registries reject the chunked variant. The fix is one line — set req.ContentLength from Stat().
  • tar.FileInfoHeader(info, "") is called with an empty link target and patched afterwards. It works, but the real signature exists for a reason: passing the target up front is what the API wants, and the manual Typeflag assignment is what you'd otherwise forget.
  • Nothing here is deterministic. Tar headers carry mtime, uid/gid, and (via Go) sub-second timestamps as PAX records; gzip output depends on the compression level and library version. Pack the same rootfs twice and you can get two different layer digests for identical content — which quietly defeats the HEAD-check dedup we just built.

💡 Fun facts

  • The config blob is also just an ordinary blob, stored by digest right alongside the layers. So the "image metadata" lives in the same content-addressable store as the gigabyte-sized layers — it just happens to be a small JSON.
  • The HEAD check before upload (mount/skip) isn't our invention, it's part of the spec: that's exactly how cross-repository blob mount works. Push a layer into library/nginx, and the same layer in myapp can be "mounted" without re-uploading the bytes.
  • That mount is a real endpoint, not a metaphor: POST /v2/<name>/blobs/uploads/?mount=<digest>&from=<other-repo>. A 201 Created means the registry linked the existing blob and moved zero bytes; a 202 Accepted means it declined and you upload the long way. When docker push prints Mounted from library/nginx, that's this call.
  • A layer has two hashes and they are never equal. The digest in the manifest is SHA256 of the compressed blob; the diffID in the config is SHA256 of the uncompressed tar. Two registries can hold the same layer content under different digests just because one recompressed it at a different gzip level — the diffIDs still match, which is exactly why the config carries them.
  • The mediaType in the manifest isn't cosmetic: it's how the registry and client tell an OCI image (vnd.oci.image.manifest.v1+json) apart from Docker schema 2 (vnd.docker.distribution.manifest.v2+json). Get it wrong and docker pull won't understand what it downloaded.
  • schemaVersion: 2 still sticks out in every manifest, even though schema 1 has been dead for years. It's the same compatibility leftover as /v2/ in the registry URL.
  • There is a famous constant blob you'll see across the ecosystem: sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a, size 2. It's the SHA256 of the two characters {}. OCI 1.1 gave it a name — application/vnd.oci.empty.v1+json — so that non-image artifacts (SBOMs, Helm charts, signatures) can fill the mandatory config slot with something meaningful-by-convention instead of inventing a payload. Every registry on earth stores that same two-byte file.
  • Pushing the same tag twice doesn't overwrite anything. The tag pointer moves to the new manifest, and the previous manifest stays in the store — unreferenced, still pullable by digest, until garbage collection runs. This is the real reason :latest is unsafe in production and digests aren't: the tag is mutable, the content isn't.
  • The spec puts a size cap on manifests — around 4 MiB — which sounds absurd until you remember a manifest is a list of pointers. If you ever hit it, you don't have a big manifest, you have thousands of layers. Docker's overlay2 driver, for its part, caps out at 128 layers.
  • gzip isn't the only option any more. OCI defines application/vnd.oci.image.layer.v1.tar+zstd, and zstd decompresses several times faster — which matters because pull latency is decompression-bound on modern networks, not bandwidth-bound. BuildKit can emit it today; the catch is that older clients don't know the media type.
  • Push order — blobs first, manifest last — is not a convention, it's the integrity model. The manifest is the only thing that makes the blobs an image, so the registry can validate the whole graph in a single check at the last step. It also means an interrupted push leaves orphaned blobs but never a half-broken image.
  • OCI 1.1 added a subject field and a Referrers API (GET /v2/<name>/referrers/<digest>) so signatures and SBOMs can be attached to an existing image without altering it. Before that, cosign faked it by pushing to a magic tag named after the digest (sha256-<hex>.sig) — a side-channel built entirely out of tag names, because tags were the only mutable thing in the spec.

What I figured out while digging into this

While writing push, it clicked that "building an image" is mostly about computing the right hashes and laying out references — not "packing files". The manifest is just JSON with a list of digests and sizes; the registry never even looks inside the layers. The trickiest part is keeping the order: blobs first, manifest last. I got it backwards once and got MANIFEST_BLOB_UNKNOWN, even though all the data was already physically on the registry's disk.

The second thing that surprised me was how much of push is status-code reading rather than data transfer. 201 versus 202 on the same request is the difference between "done" and "you're only getting started", and a client that ignores it fails three steps later with a message about a completely different object. Content-addressing makes the data model beautifully simple and pushes all the fragility into the protocol handshake.

What could be improved

  • Split the rootfs into separate layers (at least "base + diff") — then the HEAD check would actually deduplicate instead of uploading the whole image every time.
  • Fill DiffIDs with real uncompressed hashes so the manifest passes strict validation and cosign signing.
  • Check response status codes (client.Do currently ignores the error) and add chunked upload for large layers.
  • Set req.ContentLength explicitly instead of letting Go fall back to chunked encoding, and handle the 202 → PUT to Location path so the code works against registries that don't do single-request uploads.
  • Try ?mount=&from= before uploading at all — one request that either links the blob for free or tells you to send it.
  • Normalize tar headers (zero mtimes, uid/gid 0, no PAX records) so the same rootfs produces the same digest twice. Without that, dedup and reproducible builds are both theatre.
  • Trust the Docker-Content-Digest response header as the manifest's canonical digest rather than recomputing it locally.

Try it yourself

# Push to local Meadow:
sudo ./sheep tag minimal localhost:5000/myapp:v1
sudo ./sheep push localhost:5000/myapp:v1
# Verify:
curl -s localhost:5000/v2/_catalog | jq .
curl -s localhost:5000/v2/myapp/tags/list | jq .

Now the same thing entirely by hand — a complete OCI image pushed with nothing but curl, tar, and shasum:

docker run -d -p 5000:5000 --name meadow registry:2

# 1. The layer: one file, tarred and gzipped.
mkdir -p rootfs && echo hello > rootfs/hello.txt
tar -C rootfs -czf layer.tar.gz .

# Two hashes of one layer. This is the distinction from the fun facts:
LAYER_DIGEST="sha256:$(shasum -a 256 layer.tar.gz | cut -d' ' -f1)"   # compressed
DIFF_ID="sha256:$(gzip -dc layer.tar.gz | shasum -a 256 | cut -d' ' -f1)" # uncompressed
LAYER_SIZE=$(wc -c < layer.tar.gz | tr -d ' ')
echo "digest=$LAYER_DIGEST"
echo "diffID=$DIFF_ID"   # different number, same layer

# 2. The two-step upload: open a session, then PUT the bytes.
#    The Location already carries a ?_state= query, which is why the digest
#    is appended with '&' and not '?'. distribution returns it absolute;
#    the spec allows a relative path too, hence the normalisation.
LOC=$(curl -s -X POST -D - -o /dev/null \
        http://localhost:5000/v2/myapp/blobs/uploads/ \
      | tr -d '\r' | awk '/^[Ll]ocation:/ {print $2}')
case "$LOC" in /*) LOC="http://localhost:5000$LOC";; esac
curl -s -X PUT --data-binary @layer.tar.gz \
     -H "Content-Type: application/octet-stream" \
     "${LOC}&digest=${LAYER_DIGEST}" \
     -o /dev/null -w 'layer: %{http_code}\n'   # expect 201

# 3. The config blob — note diff_ids holds the UNcompressed hash.
cat > config.json <<EOF
{"architecture":"amd64","os":"linux","config":{},
 "rootfs":{"type":"layers","diff_ids":["${DIFF_ID}"]}}
EOF
CONFIG_DIGEST="sha256:$(shasum -a 256 config.json | cut -d' ' -f1)"
CONFIG_SIZE=$(wc -c < config.json | tr -d ' ')
LOC=$(curl -s -X POST -D - -o /dev/null \
        http://localhost:5000/v2/myapp/blobs/uploads/ \
      | tr -d '\r' | awk '/^[Ll]ocation:/ {print $2}')
case "$LOC" in /*) LOC="http://localhost:5000$LOC";; esac
curl -s -X PUT --data-binary @config.json \
     -H "Content-Type: application/octet-stream" \
     "${LOC}&digest=${CONFIG_DIGEST}" \
     -o /dev/null -w 'config: %{http_code}\n'

# 4. The manifest LAST — and watch the Docker-Content-Digest come back.
cat > manifest.json <<EOF
{"schemaVersion":2,
 "mediaType":"application/vnd.oci.image.manifest.v1+json",
 "config":{"mediaType":"application/vnd.oci.image.config.v1+json",
           "digest":"${CONFIG_DIGEST}","size":${CONFIG_SIZE}},
 "layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
            "digest":"${LAYER_DIGEST}","size":${LAYER_SIZE}}]}
EOF
curl -s -X PUT --data-binary @manifest.json \
     -H "Content-Type: application/vnd.oci.image.manifest.v1+json" \
     -D - -o /dev/null http://localhost:5000/v2/myapp/manifests/v1 \
  | grep -i 'HTTP/\|docker-content-digest'

# 5. A real client can now consume what curl built:
docker pull localhost:5000/myapp:v1

# Reorder the manifest above (or just pretty-print it) and push again:
# same image, different Docker-Content-Digest. The digest is over bytes.

# Cross-repository blob mount — the same layer into another repo, zero bytes:
curl -s -X POST -D - -o /dev/null \
     "http://localhost:5000/v2/other/blobs/uploads/?mount=${LAYER_DIGEST}&from=myapp" \
  | head -1
# 201 Created = mounted for free. 202 Accepted = declined, upload it the long way.

# And the famous empty blob, verified in one line:
printf '{}' | shasum -a 256
# 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a

The Image Registry series is complete. Next up — Distributed Systems Patterns.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Pull from Docker Hub | Next: Desired State vs Actual State