Lock Down Service-to-Service Traffic with mTLS via cert-manager

What you'll build / learn

You'll stand up a private certificate authority inside a Kubernetes cluster using cert-manager, then use it to issue short-lived, auto-rotating certificates to two workloads — an NGINX backend that refuses any connection without a valid client certificate, and a curl client that presents one. By the end, every request between them is mutually authenticated and encrypted, and rotation happens without you touching a thing.

The point of doing this at the pod level: a network policy tells you where traffic came from, but mTLS proves who sent it, encrypts it in transit, and — with short-lived certs — limits how long a stolen credential is worth anything. cert-manager handles the tedious parts (issuance, renewal, key rotation) declaratively, so the pattern scales past a demo.

flowchart LR
  CA[internal-ca ClusterIssuer] -- issues --> S1[Secret: backend-tls]
  CA -- issues --> S2[Secret: client-tls]
  S2 --> C[client pod / curl]
  S1 --> B[backend pod / NGINX]
  C -- "mTLS on :8443 — both sides verified" --> B

Prerequisites

Verified in August 2026 against:

• cert-manager v1.21.1 — supports Kubernetes 1.33–1.36 (v1.20 covers 1.32–1.35)
• Kubernetes v1.36 via kind v0.32.0 (kind create cluster gives you v1.36.1); any conformant cluster in the supported range works
• kubectl matching your cluster's minor version, with cluster-admin access — you'll be installing CRDs and creating ClusterIssuers
• NGINX stable-alpine image (1.30.x) and curl 8.21.0 container images, pulled automatically

macOS and Linux commands are identical. On Windows, run everything from WSL2.

1. Install cert-manager

Install the static manifest, then wait for all three deployments (controller, webhook, cainjector) to come up — the webhook must be ready before it can admit the resources you create next:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.1/cert-manager.yaml
kubectl -n cert-manager wait --for=condition=Available deployment --all --timeout=180s

2. Bootstrap a private CA

cert-manager's standard bootstrap chain is: a SelfSigned issuer signs one root CA certificate, and a CA issuer then signs everything else with it. Save as ca.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-bootstrap
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca
  # ClusterIssuers can only read Secrets from the cert-manager
  # namespace (the --cluster-resource-namespace default)
  namespace: cert-manager
spec:
  isCA: true
  commonName: internal-ca
  secretName: internal-ca-secret
  duration: 43800h # 5 years; leaf certs below are the short-lived ones
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-bootstrap
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-secret

kubectl apply -f ca.yaml
kubectl get clusterissuer internal-ca

Wait until READY shows True (the status message reads Signing CA verified).

3. Issue certificates for both workloads

Two leaf certificates: the backend's carries server auth and its Service DNS names; the client's carries client auth and an identity in its common name. Both live 24 hours and renew at the halfway mark, and rotationPolicy: Always (the default since v1.18) generates a fresh private key on every renewal. Save as certs.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: mtls-demo
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: backend-tls
  namespace: mtls-demo
spec:
  secretName: backend-tls
  duration: 24h
  renewBefore: 12h
  dnsNames:
    - backend.mtls-demo.svc
    - backend.mtls-demo.svc.cluster.local
  usages:
    - digital signature
    - key encipherment
    - server auth
  privateKey:
    rotationPolicy: Always
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: client-tls
  namespace: mtls-demo
spec:
  secretName: client-tls
  commonName: frontend
  duration: 24h
  renewBefore: 12h
  usages:
    - digital signature
    - key encipherment
    - client auth
  privateKey:
    rotationPolicy: Always
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer

kubectl apply -f certs.yaml
kubectl -n mtls-demo get certificate

Each resulting Secret holds tls.crt, tls.key, and ca.crt — that last one is what each side uses to verify the other.

4. Deploy a backend that requires client certificates

ssl_verify_client on makes NGINX reject any caller that doesn't present a certificate signed by our CA. Save as backend.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: backend-nginx
  namespace: mtls-demo
data:
  default.conf: |
    server {
      listen 8443 ssl;
      ssl_certificate        /etc/nginx/tls/tls.crt;
      ssl_certificate_key    /etc/nginx/tls/tls.key;
      ssl_client_certificate /etc/nginx/tls/ca.crt;
      ssl_verify_client      on;
      ssl_protocols          TLSv1.3;
      location / {
        return 200 "backend OK - verified client: $ssl_client_s_dn\n";
      }
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: mtls-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: nginx
          image: nginx:stable-alpine
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: tls
              mountPath: /etc/nginx/tls
              readOnly: true
            - name: conf
              mountPath: /etc/nginx/conf.d
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: backend-tls
        - name: conf
          configMap:
            name: backend-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: mtls-demo
spec:
  selector:
    app: backend
  ports:
    - port: 8443
      targetPort: 8443

kubectl apply -f backend.yaml

One caveat: NGINX loads certificates at startup. The kubelet refreshes the mounted Secret when cert-manager rotates it, but NGINX keeps serving the old (still-valid) cert until reloaded — run kubectl -n mtls-demo rollout restart deploy/backend after rotation, or automate it (see Next steps).

5. Deploy the client

A minimal pod with the client cert mounted. Save as client.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: mtls-demo
spec:
  containers:
    - name: curl
      image: curlimages/curl:8.21.0
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: tls
          mountPath: /etc/tls
          readOnly: true
  volumes:
    - name: tls
      secret:
        secretName: client-tls

kubectl apply -f client.yaml
kubectl -n mtls-demo wait --for=condition=Ready pod/client --timeout=120s

Verify it works

Call the backend with the client certificate:

kubectl -n mtls-demo exec client -- \
  curl -s --cacert /etc/tls/ca.crt \
       --cert /etc/tls/tls.crt --key /etc/tls/tls.key \
       https://backend.mtls-demo.svc.cluster.local:8443/

Expected output:

backend OK - verified client: CN=frontend

Both directions were verified: curl checked the server cert against ca.crt, and NGINX validated the client cert — $ssl_client_s_dn echoing CN=frontend proves it. Now confirm rejection without a client cert:

kubectl -n mtls-demo exec client -- \
  curl -sk https://backend.mtls-demo.svc.cluster.local:8443/

<html>
<head><title>400 No required SSL certificate was sent</title></head>
...

Finally, confirm rotation is scheduled — cert-manager will re-issue in ~12 hours, no action required:

kubectl -n mtls-demo get certificate backend-tls \
  -o jsonpath='{.status.renewalTime}{"\n"}'

To watch rotation happen live, drop duration to 1h and renewBefore to 55m and a new CertificateRequest appears every ~5 minutes (kubectl -n mtls-demo get certificaterequest).

Troubleshooting

ClusterIssuer not ready: secrets "internal-ca-secret" not found — you created the CA Certificate in your app namespace. ClusterIssuers only read Secrets from the cert-manager namespace (controlled by the controller's --cluster-resource-namespace flag). Recreate the CA Certificate with namespace: cert-manager.

curl: (60) SSL certificate problem: unable to get local issuer certificate — curl doesn't trust the backend's cert. You omitted --cacert /etc/tls/ca.crt or pointed it at a different CA than the one that signed the server cert. Both Secrets carry the correct ca.crt as long as both Certificates reference the same issuer.

curl: (60) SSL: no alternative certificate subject name matches target hostname — you're hitting the Service by a name (or IP) that isn't in the backend Certificate's dnsNames. Use the full backend.mtls-demo.svc.cluster.local form, or add the short name you're using as another SAN.

Certificate stuck with READY: False — run kubectl -n mtls-demo describe certificate <name> and read the Events. The most common cause right after install is the webhook not being ready yet (connection refused on cert-manager-webhook); re-run the kubectl wait from step 1, then re-apply.

Next steps

• Automate reloads on rotation: annotate the Deployment for Stakater Reloader, or skip Secrets entirely with the cert-manager csi-driver, which delivers per-pod certs on an in-memory volume and renews them in place.
• Distribute the CA bundle properly with trust-manager instead of copying ca.crt between namespaces.
• Swap the self-signed root for a real PKI backend — the Vault issuer and the external issuers for cloud CAs slot into the same issuerRef.
• If you need mTLS across dozens of services, a mesh like Linkerd or Istio does this transparently — cert-manager can still supply the mesh's trust anchor.