Skip to content

Pull from regestry: Auth, Manifest Lists, Multi-Arch

Pull from regestry: Auth, Manifest Lists, Multi-Arch

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

sheep pull nginx — one command, but under the hood: reference parsing, token retrieval, manifest list handling for multi-arch, layer downloading. Let's walk through each step.

Parsing the image reference

func ParseImageRef(s string) ImageRef {
    ref := ImageRef{Tag: "latest"}
    // "nginx" → registry-1.docker.io/library/nginx:latest
    // "user/repo:v2" → registry-1.docker.io/user/repo:v2
    // "ghcr.io/user/repo:tag" → ghcr.io/user/repo:tag

    if i := strings.LastIndex(s, ":"); i > 0 &&
        !strings.Contains(s[i:], "/") {
        ref.Tag = s[i+1:]
        s = s[:i]
    }

    parts := strings.Split(s, "/")
    switch {
    case len(parts) == 1:
        ref.Registry = "registry-1.docker.io"
        ref.Repo = "library/" + parts[0]
    case len(parts) == 2 && !strings.Contains(parts[0], "."):
        ref.Registry = "registry-1.docker.io"
        ref.Repo = s
    default:
        ref.Registry = parts[0]
        ref.Repo = strings.Join(parts[1:], "/")
    }
    return ref
}

nginx without slashes is a Docker Hub official image (library/nginx). user/repo without a domain is Docker Hub too. If there's a dot in the first part (ghcr.io), it's a different registry.

Auth — Bearer Token

Docker Hub requires a token for pull. Here's how to get one:

func (rc *RegistryClient) getToken(ref ImageRef) (string, error) {
    if ref.Registry == defaultRegistry {
        url := fmt.Sprintf(
            "%s?service=%s&scope=repository:%s:pull",
            dockerAuthURL, dockerService, ref.Repo)
        resp, _ := rc.client.Get(url)

        var tokenResp struct {
            Token string `json:"token"`
        }
        json.NewDecoder(resp.Body).Decode(&tokenResp)
        return tokenResp.Token, nil
    }
    return "", nil // anonymous for other registries
}

sequenceDiagram
    participant C as Sheep
    participant A as auth.docker.io
    participant R as registry-1.docker.io

    C->>A: GET /token?service=registry.docker.io&scope=repository:library/nginx:pull
    A->>C: {"token": "eyJhbGci..."}
    C->>R: GET /v2/library/nginx/manifests/latest<br/>Authorization: Bearer eyJhbGci...
    R->>C: manifest list (multi-arch)

Manifest list — multi-arch

Modern images on Docker Hub have a manifest list — an index containing manifests for different platforms:

func (rc *RegistryClient) getManifest(registryURL string,
    ref ImageRef, token string) (*manifestResponse, error) {

    accepts := []string{
        "application/vnd.oci.image.index.v1+json",
        "application/vnd.docker.distribution.manifest.list.v2+json",
        "application/vnd.oci.image.manifest.v1+json",
        "application/vnd.docker.distribution.manifest.v2+json",
    }

    body, mediaType, _ := rc.registryGet(url, token,
        strings.Join(accepts, ", "))

    if strings.Contains(mediaType, "list") ||
        strings.Contains(mediaType, "index") {
        var ml manifestList
        json.Unmarshal(body, &ml)

        // Look for linux/amd64
        digest := ""
        for _, m := range ml.Manifests {
            if m.Platform.OS == "linux" &&
                m.Platform.Architecture == "amd64" {
                digest = m.Digest
                break
            }
        }
        if digest == "" {
            digest = ml.Manifests[0].Digest
        }

        // Fetch the specific manifest
        body, _, _ = rc.registryGet(
            registryURL+"/v2/"+ref.Repo+"/manifests/"+digest,
            token, singleAccepts)
    }

    var manifest manifestResponse
    json.Unmarshal(body, &manifest)
    return &manifest, nil
}

Downloading layers

func (rc *RegistryClient) pullLayer(registryURL string,
    ref ImageRef, token, digest, rootfs string) error {
    url := fmt.Sprintf("%s/v2/%s/blobs/%s",
        registryURL, ref.Repo, digest)

    req, _ := http.NewRequest("GET", url, nil)
    if token != "" {
        req.Header.Set("Authorization", "Bearer "+token)
    }

    resp, _ := rc.client.Do(req)
    defer resp.Body.Close()

    return extractLayer(resp.Body, rootfs)
}

Each layer is a gzip-compressed tar archive. Layers are applied sequentially, each adding or removing files (via whiteout).

Result

$ sheep pull nginx
pulling manifest for nginx:latest
pulling layer 1/7 a1b2c3d4e5f6
pulling layer 2/7 f6e5d4c3b2a1
...
pulled nginx:latest (180.5 MB)

What's not ideal here

Layers are downloaded sequentially. Docker downloads them in parallel, which is significantly faster for images with many layers. There's also no layer caching between pulls of different tags of the same image.

  • The token is anonymous and short-lived. On a large image it can expire mid-pull — and the registry returns 401 halfway through downloading layers. Our client doesn't handle this: you'd need to catch the 401 and re-fetch the token.
  • The linux/amd64 choice is hardcoded. On an ARM machine (Apple Silicon, Graviton) this silently downloads the wrong architecture, and the container dies with exec format error instead of a clear message.
  • The auth server address is hardcoded for Docker Hub. Per the spec you aren't supposed to know it up front: the registry returns a 401 with a WWW-Authenticate: Bearer realm="...",service="...",scope="..." header, and the client is meant to follow that realm. Our code works against Docker Hub and silently falls back to anonymous everywhere else, so it can't pull from a private GHCR or ECR at all.
  • Matching the platform on OS and Architecture alone is incomplete. The index also carries variant (v7 for 32-bit ARM, v8 for arm64) and os.version for Windows. Without variant, 32-bit ARM can pick the wrong image and land back on exec format error.
  • The digest = ml.Manifests[0].Digest fallback looks safe and isn't: modern indexes sit images next to attestation manifests (SBOM, provenance) with platform: unknown/unknown. Instead of "grab the first one if ours isn't there", it should either return an honest error or at minimum filter out unknown.

💡 Fun facts

  • docker pull nginx never talks to docker.io directly — it goes to registry-1.docker.io, and for the token to a separate auth.docker.io. docker.io itself is just a website. Splitting the auth server from the registry is part of the Bearer token scheme from RFC 6750.
  • The library/ prefix for official images is a historical leftover. That's why nginx is really library/nginx, and exactly why the code has a separate branch for an image with no slashes.
  • The manifest list (OCI image index) is why a single tag nginx:latest works on x86, ARM, and even Windows. The client picks the right digest for the platform itself; the "image" behind a tag is really a list of images.
  • Docker Hub introduced rate limits on anonymous pulls (November 2020) — which is exactly why the anonymous token we get for free is counted per IP. A lot of CI failures "out of nowhere" are just an exhausted pull limit: one office NAT or one runner pool means one IP for everybody behind it.
  • The token we get back is a plain JWT, and you can decode it. Inside is an access claim spelling out the exact grant — and, best of all, the limits themselves: "pull_limit":"100", "pull_limit_interval":"21600". So an anonymous token isn't just counted per IP; it carries its own quota, 100 pulls per 6 hours, signed by the registry.
  • Not every entry in a manifest list is an image. In nginx:latest, half the entries have platform: {os: unknown, architecture: unknown} and an annotation vnd.docker.reference.type: attestation-manifest — SBOMs and provenance added by BuildKit. Which makes our ml.Manifests[0].Digest fallback a landmine: if the platform we need isn't in the index, it can pick an attestation instead of an image, and that's something you fundamentally cannot run.
  • Docker Hub doesn't serve blobs itself: GET /blobs/ answers with a 307 to a CDN, with a signed URL in the query string. And here's the subtlety that makes our naive code work: since Go 1.8, the HTTP client drops the Authorization header when a redirect crosses to another host. If it forwarded it, the CDN would reject the request — two auth mechanisms in one call.
  • Windows images have layers the registry never stores. The manifest marks them foreign (nondistributable) and gives them a urls field — the client fetches them straight from Microsoft's servers, because licensing doesn't let the registry redistribute them. One manifest format, bent around a legal constraint.
  • Deleting a file in a layer is really creating a file. A layer can't remove anything, so OCI encodes a deletion as a whiteout file .wh.<name>, and clearing a whole directory as .wh..wh..opq. Which is why an image that "deleted" a secret in a later layer still carries it in the earlier one.
  • Early Docker Registry v2 signed manifests with JWS (schema 1), and the digest had to be computed over a canonicalized form — any JSON reformatting broke the signature. Schema 2 threw signatures out entirely and leans on the same content-addressing we walked through in the previous part.

What I figured out while digging into this

For years I assumed an "image" was a single file. But once I started parsing the Docker Hub response, it clicked: the tag points to a manifest list, that points to a per-platform manifest, that points to a config blob and a list of layer digests. Four levels of indirection, and no layer knows its own tag. The tag is just a movable pointer at the very top, and everything underneath is immutable and hash-addressed.

What could be improved

  • Download layers in parallel via an errgroup with bounded concurrency — the biggest win for multi-layer images.
  • Detect the platform via runtime.GOOS/runtime.GOARCH instead of a hardcoded linux/amd64, and return a clear error if the needed one isn't in the index.
  • Add retry with backoff and 401 handling (token re-fetch) — without it, pulling large images is flaky.
  • Discover the auth server from the WWW-Authenticate header after the first 401 instead of hardcoding the URL — the same code then starts working against GHCR, ECR, and private registries.
  • Verify the digest of every downloaded layer instead of trusting the registry: hash it on the fly with io.MultiWriter (as in the previous part) and compare against the manifest.
  • Cache layers by digest in a shared blob store instead of extracting into each image's rootfs separately — then nginx:1.25 and nginx:1.26 reuse their common layers.

Try it yourself

# Pull nginx from Docker Hub:
sudo ./sheep pull nginx
sudo ./sheep pull alpine:3.19
sudo ./sheep images
# Check the rootfs:
ls /var/lib/sheep/images/*/rootfs/ | head -10

# Now the same dance by hand, without Docker — the token first:
TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/nginx:pull" | jq -r .token)

# What's actually inside that token. A JWT is three dot-separated base64url parts,
# so first translate to plain base64 and add the padding back:
P=$(echo "$TOKEN" | cut -d. -f2 | tr '_-' '/+')
printf '%s' "$P$(printf '=%.0s' $(seq $(( (4 - ${#P} % 4) % 4 ))))" | base64 -d | jq .access
# Look at `parameters`: pull_limit and pull_limit_interval — your quota ships inside the token.

# The manifest list: how many platforms live behind one `latest` tag —
# and which entries aren't images at all, but attestations:
curl -s -H "Authorization: Bearer $TOKEN" \
     -H "Accept: application/vnd.oci.image.index.v1+json" \
     https://registry-1.docker.io/v2/library/nginx/manifests/latest \
  | jq -c '.manifests[] | {platform, type: .annotations["vnd.docker.reference.type"]}'

Pull works. Next up — Push: how to create an OCI image and send it to your own registry.

Resources

Source code for the series: github.com/igorgorovoy/sheep-shepherd-meadow

Previous: Content-Addressable Storage | Next: Push to Your Own Registry