Autoscale GPU Inference on EKS with Karpenter and Spot Instances

What you'll build

A Kubernetes-hosted inference stack on Amazon EKS where Karpenter provisions spot GPU nodes on demand, bin-packs a vLLM model server onto the cheapest available capacity, and deletes the nodes when traffic drops — no static GPU node group burning money at 3 a.m.

Prerequisites

Verified against Karpenter v1.14.0, Kubernetes 1.36 on EKS, NVIDIA device plugin v0.19.3, and vLLM v0.25.1 in July 2026. You need:

• An EKS cluster (Kubernetes 1.33+) with Karpenter v1.14.0 already installed per the official getting-started guide — that guide sets up the KarpenterNodeRole-<cluster> IAM role, the spot-interruption SQS queue (settings.interruptionQueue), and karpenter.sh/discovery tags on your subnets and security groups. This tutorial assumes all of that exists.
• kubectl, Helm 3.14+, and AWS CLI v2 authenticated against the cluster's account.
• A region with G5/G6 capacity (e.g. us-east-1, us-west-2).
• An EC2 spot quota above zero for GPU instances — check All G and VT Spot Instance Requests (quota code L-3819A6DF). Fresh accounts often have 0, which blocks everything below:

aws service-quotas get-service-quota \
  --service-code ec2 --quota-code L-3819A6DF \
  --query 'Quota.Value'

If it returns 0.0, request an increase (8+ vCPUs) before starting. Export your cluster name for the steps below:

export CLUSTER_NAME=<your-cluster>

1. Enable spot in the account

If this account has never launched a spot instance, EC2 needs its service-linked role:

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

An InvalidInput error saying the role name "has been taken" means it already exists — safe to ignore.

2. Create a GPU EC2NodeClass

The EC2NodeClass tells Karpenter how to build GPU nodes. The al2023@latest AMI alias is doing real work here: Karpenter resolves it to the EKS-optimized AL2023 AMI variants — including the NVIDIA accelerated one, which ships the GPU drivers and container toolkit — and picks the right variant per instance type automatically. The 100 GiB root volume matters because the vLLM image alone is over 10 GiB; the AL2023 default of 20 GiB gets you disk-pressure evictions.

cat <<EOF | kubectl apply -f -
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu-inference
spec:
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
EOF

3. Create the spot GPU NodePool

The NodePool defines what Karpenter may launch. Two families (g6 = NVIDIA L4, g5 = A10G) across all sizes gives EC2 Fleet many spot pools to choose from — diversification is what keeps spot interruption rates low. Allowing both capacity types means Karpenter uses spot when it's available and falls back to on-demand only when it isn't; Fleet picks the actual pool with the price-capacity-optimized strategy (lowest price, lowest interruption risk). The taint keeps non-GPU workloads off these expensive nodes, and the limits block caps the blast radius at 8 GPUs no matter what gets deployed.

cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      taints:
        - key: nvidia.com/gpu
          value: "true"
          effect: NoSchedule
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["g6", "g5"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-inference
  limits:
    nvidia.com/gpu: "8"
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
EOF

consolidateAfter: 5m gives you five minutes of headroom before Karpenter starts tearing down underutilized nodes — long enough to survive a rolling restart without churning instances.

4. Install the NVIDIA device plugin

The accelerated AMI has drivers, but Kubernetes only learns about the GPUs from the NVIDIA device plugin. Karpenter also won't consider a GPU node fully initialized until the nvidia.com/gpu resource registers, so this DaemonSet is load-bearing. Its default values already tolerate the nvidia.com/gpu:NoSchedule taint:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
helm upgrade -i nvdp nvdp/nvidia-device-plugin \
  --namespace nvidia-device-plugin --create-namespace \
  --version 0.19.3

5. Deploy the model server

A vLLM deployment serving Qwen2.5-0.5B-Instruct — small enough to download in a minute and fit any single L4/A10G. The nvidia.com/gpu: "1" request is the trigger: the pod goes Pending, Karpenter sees an unschedulable GPU pod, and launches a node for it. The in-memory /dev/shm volume avoids vLLM's shared-memory warnings; the CPU/memory requests are sized so one replica bin-packs cleanly onto an xlarge.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.25.1
          args:
            - --model=Qwen/Qwen2.5-0.5B-Instruct
            - --max-model-len=4096
          ports:
            - containerPort: 8000
          resources:
            requests:
              cpu: "3"
              memory: 10Gi
              nvidia.com/gpu: "1"
            limits:
              nvidia.com/gpu: "1"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
          volumeMounts:
            - name: shm
              mountPath: /dev/shm
      volumes:
        - name: shm
          emptyDir:
            medium: Memory
            sizeLimit: 2Gi
EOF

Verify it works

Watch Karpenter react — a NodeClaim should appear within seconds and go Ready in about two minutes:

kubectl get nodeclaims -w

NAME                  TYPE        CAPACITY   ZONE         NODE                          READY   AGE
gpu-inference-8xkzt   g6.xlarge   spot       us-east-1b   ip-192-168-73-21.ec2.internal True    2m4s

CAPACITY: spot is the money column. Once the pod is Ready (first start takes a few minutes: image pull plus model download), hit the OpenAI-compatible API:

kubectl port-forward deploy/vllm 8000:8000 &
curl -s localhost:8000/v1/models

Expected output (trimmed):

{"object":"list","data":[{"id":"Qwen/Qwen2.5-0.5B-Instruct","object":"model","owned_by":"vllm", ...}]}

Now watch the bin-packing. Scale up and Karpenter batches the pending pods, then asks EC2 Fleet for the cheapest combination that fits them — several single-GPU spot nodes or one multi-GPU node, whichever prices lower:

kubectl scale deploy/vllm --replicas=3
kubectl get nodeclaims

Scale back to 1 and, five minutes later (consolidateAfter), watch Karpenter consolidate the now-empty nodes out of existence. That's the whole cost story: GPU capacity exists only while pods need it, and it's spot-priced — AWS quotes up to 90% off on-demand, and G-family discounts routinely land north of 50%.

Troubleshooting

Karpenter logs show MaxSpotInstanceCountExceeded during instance launch — your spot vCPU quota for G instances is 0 or too low. Request an increase on L-3819A6DF (All G and VT Spot Instance Requests); until it lands, Karpenter will retry and may fall back to on-demand.

Pod event: incompatible with nodepool "gpu-inference" ... no instance type satisfied resources — the pod's requests can't fit any instance the NodePool allows, or the NodePool hit its limits. Check that requests fit an xlarge (4 vCPU/16 GiB) and that existing NodeClaims haven't consumed the 8-GPU cap.

Node is Ready but the pod stays Pending with Insufficient nvidia.com/gpu — the device plugin isn't running on the node. kubectl get pods -n nvidia-device-plugin -o wide should show one pod per GPU node; if you overrode chart values, make sure the DaemonSet still tolerates nvidia.com/gpu:NoSchedule.

vLLM CrashLoopBackOff with torch.OutOfMemoryError: CUDA out of memory — the model plus KV cache doesn't fit the GPU. Lower --max-model-len, pass --gpu-memory-utilization 0.85, or constrain the NodePool to instance types with more VRAM.

Next steps

Pin the AMI alias to a specific version (al2023@v2026...) in production — @latest triggers drift replacement of GPU nodes on every AMI release, which can churn long-running inference pods. Add a second, weighted NodePool with on-demand only to guarantee a baseline while spot handles overflow. Karpenter only scales nodes; pair it with an HPA or KEDA scaling on queue depth or request latency so replicas track traffic. From there, dig into disruption budgets to bound how many nodes consolidate at once, and NVIDIA time-slicing or MIG in the device plugin to pack multiple small models onto one GPU.