Autoscale GKE Node Pools with Cluster Autoscaler and Spot VMs

What you'll build

A GKE Standard cluster with two node pools: a small on-demand pool for system workloads, and an autoscaling Spot VM pool that scales from zero to N nodes based on pending pods. You'll taint the Spot pool so only workloads that explicitly tolerate interruption land there, and configure the autoscaler to scale down aggressively when idle.

Prerequisites

• gcloud CLI (SDK 450.0.0+) authenticated against a GCP project with billing enabled and the Kubernetes Engine API turned on
• kubectl matching your target cluster's minor version, plus the auth plugin: gcloud components install gke-gcloud-auth-plugin
• IAM role roles/container.admin (or equivalent) on the project
• A GKE Standard cluster (not Autopilot, node pool management is manual here)
• Basic familiarity with taints, tolerations, and PodDisruptionBudgets

Set a couple of variables you'll reuse:

export PROJECT_ID=$(gcloud config get-value project)
export REGION=us-central1
export CLUSTER=prod-cluster

1. Create the cluster with a small on-demand system pool

Keep kube-system pods, ingress controllers, and anything stateful off Spot capacity. A regional cluster spreads nodes across three zones by default, so --num-nodes 1 actually gives you three nodes total, one per zone.

gcloud container clusters create $CLUSTER \
  --region=$REGION \
  --num-nodes=1 \
  --machine-type=e2-standard-2 \
  --release-channel=regular

Grab credentials once it's up:

gcloud container clusters get-credentials $CLUSTER --region=$REGION

2. Add an autoscaling Spot VM node pool

This is the pool that actually saves you money. --min-nodes 0 lets it scale to zero when there's nothing to run.

gcloud container node-pools create spot-pool \
  --cluster=$CLUSTER \
  --region=$REGION \
  --spot \
  --machine-type=e2-standard-4 \
  --num-nodes=0 \
  --enable-autoscaling \
  --min-nodes=0 \
  --max-nodes=6 \
  --location-policy=ANY \
  --node-taints=cloud.google.com/gke-spot=true:NoSchedule

A few things worth knowing here:

• --min-nodes/--max-nodes on a regional pool are per zone, not cluster-wide. Three zones times --max-nodes=6 means the cluster can burst to 18 spot nodes, not 6. Size accordingly.
• --location-policy=ANY tells the autoscaler to prefer zones with more available Spot capacity (and historically lower preemption rates) rather than balancing evenly. For Spot pools this is almost always what you want; BALANCED (the default) is better for on-demand pools where even distribution matters more than availability.
• Spot VMs replaced Preemptible VMs a few years back. The practical difference: Spot has no 24-hour forced termination, dynamic pricing instead of a flat 60-91% discount, and the same 30-second preemption notice.
• GKE automatically labels every node in this pool cloud.google.com/gke-spot="true". It does not automatically taint them, that's on you, which is why the --node-taints flag matters. Skip it and regular pods can land on Spot nodes and get evicted without warning.

3. Schedule workloads onto Spot with tolerations and grace periods

Only pods that explicitly tolerate the taint will get scheduled on the Spot pool. Pair that with a short terminationGracePeriodSeconds (Spot gives you ~30 seconds of SIGTERM warning before reclaim) and a PodDisruptionBudget so voluntary disruptions don't wipe out too many replicas at once.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: batch-worker
spec:
  replicas: 6
  selector:
    matchLabels:
      app: batch-worker
  template:
    metadata:
      labels:
        app: batch-worker
    spec:
      terminationGracePeriodSeconds: 25
      tolerations:
        - key: cloud.google.com/gke-spot
          operator: Equal
          value: "true"
          effect: NoSchedule
      nodeSelector:
        cloud.google.com/gke-spot: "true"
      containers:
        - name: worker
          image: gcr.io/google-samples/hello-app:2.0
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: batch-worker-pdb
spec:
  minAvailable: 4
  selector:
    matchLabels:
      app: batch-worker

kubectl apply -f batch-worker.yaml

With zero Spot nodes running, these pods will sit Pending until the autoscaler notices and provisions capacity, usually within a couple of minutes.

4. Tune the autoscaler's scale-down behavior

The default autoscaling profile (balanced) is conservative about removing nodes. For bursty batch workloads on Spot, optimize-utilization scales down faster and packs pods tighter, which matters more when you're paying by the node-hour:

gcloud container clusters update $CLUSTER \
  --region=$REGION \
  --autoscaling-profile=optimize-utilization

This is a cluster-wide setting, it affects every pool's scale-down decisions, not just Spot.

Verify it works

Check the pool is registered and empty at rest:

gcloud container node-pools describe spot-pool --cluster=$CLUSTER --region=$REGION \
  --format='value(autoscaling.minNodeCount,autoscaling.maxNodeCount)'

Force a scale-up by pushing replica count past current capacity:

kubectl scale deployment batch-worker --replicas=40
kubectl get nodes -l cloud.google.com/gke-spot=true --watch

You should see new nodes appear within roughly 1-3 minutes, each carrying the taint and label. Confirm pods actually landed there:

kubectl get pods -o wide -l app=batch-worker

Scale back down and watch nodes disappear after the idle timeout (default 10 minutes of underutilization):

kubectl scale deployment batch-worker --replicas=0
kubectl get nodes -l cloud.google.com/gke-spot=true --watch

To see savings, filter Cloud Billing reports by SKU description containing "Spot Preemptible" and compare against what the same machine type costs on-demand, usually a 60-91% reduction depending on region and machine family.

Troubleshooting

Pods stuck Pending with no node events. Usually a toleration/taint mismatch, double-check the taint key, value, and effect match exactly between the node pool and the pod spec. Also check for regional Spot capacity exhaustion, symptoms show up as scale-up events that fail silently. Try a different machine type or region, or drop --location-policy=ANY in favor of letting the autoscaler try harder across zones.

Nodes never scale down. Inspect the autoscaler's own status:

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

Common blockers: a PodDisruptionBudget that can't tolerate any eviction, pods with cluster-autoscaler.kubernetes.io/safe-to-evict: "false", or pods using local storage that the autoscaler refuses to move.

Frequent restarts from preemption. Spread replicas with topologySpreadConstraints across zones, keep terminationGracePeriodSeconds under 30, and make sure your PDB's minAvailable isn't too aggressive relative to preemption frequency.

kubectl auth errors like "exec plugin: invalid apiVersion". Install the auth plugin and set the environment variable:

gcloud components install gke-gcloud-auth-plugin
export USE_GKE_GCLOUD_AUTH_PLUGIN=True

Next steps

Look at GKE Autopilot's built-in Spot support (nodeSelector: cloud.google.com/gke-spot: "true" with no node pool management at all) if you want less infrastructure to own. Pair this setup with the Horizontal Pod Autoscaler so replica count itself reacts to load, not just node availability. And if preemption handling needs custom logic beyond SIGTERM, look at running a DaemonSet that watches the GCE metadata server's maintenance-event endpoint for early preemption signals.