Skip to content

Bridge Networking: Giving a Container an IP Address

Bridge Networking: Giving a Container an IP Address

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

A container in a new network namespace has no network. None at all. Even loopback isn't up. We need to create a virtual network and connect the container to the host. Here's how it works in Sheep.

Network architecture

graph TB
    subgraph "Host"
        ETH["eth0<br/>192.168.1.x"]
        BR["sheep0 (bridge)<br/>10.20.0.1/16"]
        VETH1H["veth_abc1"]
        VETH2H["veth_def2"]
        IPTABLES["iptables NAT<br/>MASQUERADE"]
    end

    subgraph "Container 1"
        VETH1G["eth0<br/>10.20.0.2/16"]
    end

    subgraph "Container 2"
        VETH2G["eth0<br/>10.20.0.3/16"]
    end

    BR --- VETH1H
    BR --- VETH2H
    VETH1H -.- VETH1G
    VETH2H -.- VETH2G
    BR --- IPTABLES
    IPTABLES --- ETH

Three components: a bridge (virtual switch), veth pairs (virtual cables), NAT (internet access).

Constants

const (
    BridgeName    = "sheep0"
    BridgeSubnet  = "10.20.0.0/16"
    BridgeGateway = "10.20.0.1"
)

var ipCounter uint32 = 1

The 10.20.0.0/16 subnet gives 65534 addresses. For a learning project -- more than enough.

Creating the bridge

func ensureBridge() error {
    // Check if the bridge already exists
    if _, err := net.InterfaceByName(BridgeName); err == nil {
        return nil
    }

    // Create the bridge
    run("ip", "link", "add", BridgeName, "type", "bridge")
    run("ip", "addr", "add", BridgeGateway+"/16", "dev", BridgeName)
    run("ip", "link", "set", BridgeName, "up")

    // Enable IP forwarding
    os.WriteFile("/proc/sys/net/ipv4/ip_forward", []byte("1"), 0644)

    // NAT for outgoing traffic
    run("iptables", "-t", "nat", "-A", "POSTROUTING",
        "-s", BridgeSubnet, "-j", "MASQUERADE")

    return nil
}

The bridge is created once with the first container. ip_forward lets the kernel forward packets between interfaces.

Setting up container networking

Here's the full function:

func setupNetworkForContainer(c *Container, pid int) error {
    if err := ensureBridge(); err != nil {
        return fmt.Errorf("ensure bridge: %w", err)
    }

    ip := allocateIP()

    vethHost := fmt.Sprintf("veth%s", ShortID(c.ID)[:8])
    vethGuest := "eth0"

    // Create veth pair
    run("ip", "link", "add", vethHost,
        "type", "veth", "peer", "name", vethGuest)

    // Host end goes into the bridge
    run("ip", "link", "set", vethHost, "master", BridgeName)

    // Guest end goes into the container's namespace
    run("ip", "link", "set", vethGuest,
        "netns", strconv.Itoa(pid))

    // Bring up the host end
    run("ip", "link", "set", vethHost, "up")

    // Configure networking inside the container
    nsRun(pid, "ip", "addr", "add", ip+"/16", "dev", "eth0")
    nsRun(pid, "ip", "link", "set", "eth0", "up")
    nsRun(pid, "ip", "link", "set", "lo", "up")
    nsRun(pid, "ip", "route", "add", "default", "via", BridgeGateway)

    c.Network = &NetworkSettings{
        IPAddress: ip,
        Gateway:   BridgeGateway,
        Bridge:    BridgeName,
        VethHost:  vethHost,
        VethGuest: vethGuest,
    }

    return nil
}

Veth pairs -- a virtual cable

Veth (virtual ethernet) is a pair of network interfaces connected to each other. Whatever enters one end comes out the other. One end stays on the host (attached to the bridge), the other is moved into the container's network namespace.

// nsRun executes a command inside the network namespace of a process
func nsRun(pid int, name string, args ...string) error {
    nsArgs := append([]string{
        "-t", strconv.Itoa(pid), "-n", "--", name,
    }, args...)
    return run("nsenter", nsArgs...)
}

nsenter is a utility that "enters" the namespace of a process with a given PID. The -n flag means the network namespace.

IP allocation

func allocateIP() string {
    n := atomic.AddUint32(&ipCounter, 1)
    return fmt.Sprintf("10.20.%d.%d", (n>>8)&0xFF, n&0xFF)
}

A simple atomic counter. The IP is persisted to disk:

func LoadIPCounter(baseDir string) {
    data, _ := os.ReadFile(
        filepath.Join(baseDir, "network", "ip_counter"))
    if n, err := strconv.ParseUint(
        strings.TrimSpace(string(data)), 10, 32); err == nil {
        atomic.StoreUint32(&ipCounter, uint32(n))
    }
}

Where this breaks

Our IP allocation doesn't account for released addresses. The counter only goes up. Docker has a full IPAM (IP Address Management) that tracks the pool and reuses addresses.

The second thing -- we use ip and nsenter as external commands. Docker and containerd use netlink through a Go library, which is faster and doesn't depend on installed utilities. This is on the roadmap to rewrite — the task is already in the Sheep backlog.

Try it yourself

sudo ./sheep run --name net-test -d minimal /bin/sleep 3600
# On the host, check the bridge and veth:
ip link show type bridge
ip link show type veth
# Container IP:
sudo ./sheep inspect net-test | grep IP

The container has an IP. Next up -- NAT and iptables: how packets reach the internet and come back.

Resources

Previous: OverlayFS | Next: NAT and iptables