Skip to content

Internal TLS Communication Between Pods in Kubernetes using cert-manager

Written by:

Igor Gorovyy
DevOps Engineer Lead & Senior Solutions Architect

LinkedIn

w

Why cert-manager is critical for Kubernetes

In today's world of DevSecOps and zero-trust architecture, securing internal communication between microservices becomes not just a recommendation, but a mandatory requirement. cert-manager solves one of the most complex problems in Kubernetes cluster management — automating the TLS certificate lifecycle.

Key problems cert-manager solves:

  • Manual certificate management: without automation, administrators are forced to manually generate, renew, and distribute certificates
  • Outages due to expired certificates: human factor often leads to untimely certificate renewals
  • Security inconsistency: different teams may use different approaches to TLS
  • Scaling complexity: as the number of services increases, certificate management becomes uncontrollable

Key benefits of cert-manager:

  1. Full automation: automatic generation, renewal, and distribution of certificates
  2. Support for various Issuers: from self-signed to Let's Encrypt and enterprise CAs
  3. Kubernetes-native: integration through CRD (Custom Resource Definitions)
  4. High security level: adherence to TLS best practices
  5. Reduced operational costs: minimizing manual DevOps team work
  6. Compliance: automatic adherence to corporate security policies

This document describes how to configure internal TLS or mTLS communication between pods in Kubernetes using self-signed certificates through cert-manager.

Key cert-manager Concepts and Interaction Architecture

graph LR
    A[Certificate] -->|CSR| B[CertificateRequest]
    B -->|sent to| C[Issuer / ClusterIssuer]
    C -->|signs| D[CertificateRequest]
    D -->|issues| E[Secret]
    E -->|contains| F[tls.crt / tls.key / ca.crt]
    F -->|mounted to| G[Pod]
    C --> H[CA / SelfSigned / ACME]
  • Issuer / ClusterIssuer: objects that define certificate sources (e.g., SelfSigned, CA, ACME). Issuer operates within a namespace, ClusterIssuer — globally.
  • Certificate: describes the desired TLS certificate for a service. cert-manager automatically creates the certificate and places it in a Secret.
  • CertificateRequest: internal object that cert-manager generates when processing a Certificate. Contains CSR request.
  • Secret: Kubernetes object where the ready certificate is stored (tls.crt, tls.key, ca.crt).
  • SelfSigned / CA Issuer: certificate sources that don't require external services. Suitable for internal infrastructure.
  • Renewal: automatic certificate renewal before expiration (typically 30 days before).
  • Solver (for ACME): domain ownership verification mechanism (HTTP01, DNS01). Not relevant for SelfSigned/CA.

Action Algorithm

  1. Create SelfSigned Issuer for root CA generation.
  2. Generate root CA certificate, which will be the trusted certificate authority.
  3. Create CA Issuer based on root CA.
  4. Generate TLS certificates for each service, which will be signed by CA.
  5. Connect certificates to Deployments, so services can use TLS.
  6. Configure HTTPS/HTTP client and server using certificates.
  7. (Optional) Configure mutual authentication (mTLS).

Process Flow

graph LR
    A1[SelfSigned Issuer] --> A2[Root CA Certificate]
    A2 --> B1[CA Issuer]
    B1 --> C1[service-a TLS cert]
    B1 --> C2[service-b TLS cert]
    C1 -->|Mounted as Secret| D1[service-a Pod]
    C2 -->|Mounted as Secret| D2[service-b Pod]
    D1 -->|HTTPS / mTLS| D2

📄 YAML Manifests

1. SelfSigned Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: default
spec:
  selfSigned: {}

2. Root CA Certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: root-ca
  namespace: default
spec:
  isCA: true
  commonName: my-org-root-ca
  secretName: root-ca-secret
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer

3. CA Issuer based on root CA

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: ca-issuer
  namespace: default
spec:
  ca:
    secretName: root-ca-secret

4. Certificate for service-a

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: service-a-cert
  namespace: default
spec:
  secretName: service-a-tls
  duration: 8760h
  renewBefore: 720h
  commonName: service-a.default.svc.cluster.local
  dnsNames:
    - service-a
    - service-a.default
    - service-a.default.svc
    - service-a.default.svc.cluster.local
  issuerRef:
    name: ca-issuer
    kind: Issuer

5. Certificate for service-b

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: service-b-cert
  namespace: default
spec:
  secretName: service-b-tls
  duration: 8760h
  renewBefore: 720h
  commonName: service-b.default.svc.cluster.local
  dnsNames:
    - service-b
    - service-b.default
    - service-b.default.svc
    - service-b.default.svc.cluster.local
  issuerRef:
    name: ca-issuer
    kind: Issuer

6. Volume in Deployment

volumeMounts:
  - name: tls-cert
    mountPath: /etc/tls
    readOnly: true

volumes:
  - name: tls-cert
    secret:
      secretName: service-a-tls

How It Works 🚀

  1. SelfSigned Issuer generates a CA certificate (root-ca), stored in root-ca-secret.
  2. CA Issuer uses root-ca to sign other certificates.

  3. For each service cert-manager:

  4. creates CSR

  5. signs through ca-issuer

  6. stores result in Secret (*.tls), which includes tls.crt, tls.key, ca.crt

  7. TLS certificate is mounted to pod via volume + mountPath — detailed process:

Step 4.1: Volume Definition in Deployment 

volumes:
- name: tls-cert                    # Volume name
  secret:
    secretName: service-a-tls       # Reference to Secret with certificate
    defaultMode: 0400               # Access permissions (read-only for owner)
    items:                          # (Optional) selective file mounting
    - key: tls.crt
      path: server.crt
    - key: tls.key
      path: server.key
    - key: ca.crt
      path: ca.crt

Step 4.2: Container Mount 

volumeMounts:
- name: tls-cert                    # Volume name (must match volumes.name)
  mountPath: /etc/tls               # Path in container filesystem
  readOnly: true                    # Mandatory for security
  subPath: ""                       # (Optional) subfolder mounting

Step 4.3: Result in Container After mounting, files appear in the container: 

/etc/tls/
├── tls.crt          # Public certificate (PEM format)
├── tls.key          # Private key (PEM format)  
└── ca.crt           # Certificate authority certificate

Step 4.4: Internal Kubernetes Mechanism

graph TD
    A[kubelet] -->|Request Secret| B[Kubernetes API Server]
    B -->|RBAC Check| C[etcd]
    C -->|Returns Secret data| B
    B -->|Passes to kubelet| A
    A -->|Creates tmpfs| D[In-Memory Volume]
    D -->|Mounts files| E[Container Filesystem]

    F[Secret Update] -->|Automatic| G[Volume Refresh]
    H[Pod Restart] -->|Cleanup| I[tmpfs Cleared]

Step 4.5: Security Aspects of Mounting
* tmpfs storage: Certificates are stored in RAM, not on disk
* Automatic cleanup: When Pod is deleted, certificates automatically disappear
* RBAC validation: kubelet checks ServiceAccount permissions to read Secret
* Namespace isolation: Secret is only accessible within its namespace
* File permissions: defaultMode sets secure access permissions (0400 = read-only for owner)

Step 4.6: Practical Code Usage 

// Go example of certificate reading
cert, err := tls.LoadX509KeyPair("/etc/tls/tls.crt", "/etc/tls/tls.key")
if err != nil {
    log.Fatal(err)
}

// HTTPS server configuration
server := &http.Server{
    Addr:      ":8443",
    TLSConfig: &tls.Config{Certificates: []tls.Certificate{cert}},
}


// Node.js example using certificates
const https = require('https');
const fs = require('fs');
const express = require('express');

const app = express();

// Reading certificates
const options = {
  key: fs.readFileSync('/etc/tls/tls.key'),
  cert: fs.readFileSync('/etc/tls/tls.crt'),
  ca: fs.readFileSync('/etc/tls/ca.crt'),
  requestCert: true,        // For mTLS
  rejectUnauthorized: true  // Strict certificate validation
};

app.get('/', (req, res) => {
  res.send('Secure HTTPS server with cert-manager certificates!');
});

// Creating HTTPS server
https.createServer(options, app).listen(8443, () => {
  console.log('HTTPS Server running on port 8443');
});

// Example HTTPS client for mTLS
const clientOptions = {
  hostname: 'service-b.default.svc.cluster.local',
  port: 8443,
  path: '/',
  method: 'GET',
  key: fs.readFileSync('/etc/tls/tls.key'),
  cert: fs.readFileSync('/etc/tls/tls.crt'),
  ca: fs.readFileSync('/etc/tls/ca.crt')
};

const req = https.request(clientOptions, (res) => {
  console.log('Status:', res.statusCode);
  res.on('data', (data) => {
    console.log(data.toString());
  });
});

req.end();
  1. Applications use certificates to run HTTPS servers or make requests via HTTPS
  2. With mTLS: both sides authenticate each other based on ca.crt

TLS vs mTLS

TLS (one-way) mTLS (mutual)
Server has certificate ✅ ✅
Client has certificate ❌ ✅
Client verification ❌ ✅