Fan Out Scientific Computing Jobs on AWS Batch with Spot Instances

What you'll build

A compute-heavy Monte Carlo simulation packaged in a Docker container and fanned out as a 250-way AWS Batch array job running entirely on EC2 Spot Instances — an architecture that scales to thousands of vCPUs at roughly a third of On-Demand price, and scales back to zero when the queue is empty.

Prerequisites

• An AWS account with admin-level credentials and a default VPC in your region (every account created since 2013 has one).
• AWS CLI v2 — verified against 2.35.x. Check with aws --version.
• Docker Engine — verified against 29.7. Check with docker --version.
• Commands verified against AWS Batch docs, August 2026. Tested in us-east-1; any commercial region works.

Cost note: the compute environment below has minvCpus: 0, so you pay only while jobs run. The full 250-task run costs well under a dollar on Spot.

Set these variables in your shell — every command below uses them:

export AWS_REGION=us-east-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export ECR_URI=$ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/pi-sim

1. Write the simulation and containerize it

Any embarrassingly parallel workload fits this pattern. We'll estimate π by throwing random darts — each array task gets a unique AWS_BATCH_JOB_ARRAY_INDEX from Batch (0 to N−1), which we use as the RNG seed so shards don't duplicate work.

sim.py:

import os, random, time

shard = int(os.environ.get("AWS_BATCH_JOB_ARRAY_INDEX", 0))
samples = int(os.environ.get("SAMPLES", 10_000_000))
random.seed(shard)

start = time.time()
hits = sum(1 for _ in range(samples)
           if random.random() ** 2 + random.random() ** 2 <= 1.0)
pi = 4 * hits / samples
print(f"shard={shard} samples={samples} pi_estimate={pi:.6f} secs={time.time() - start:.1f}")

Dockerfile:

FROM python:3.13-slim
COPY sim.py /app/sim.py
CMD ["python", "/app/sim.py"]

Build for linux/amd64 explicitly — on Apple Silicon a default build produces an arm64 image that x86 Batch instances can't run:

docker build --platform linux/amd64 -t pi-sim .

2. Push the image to ECR

Create a private Amazon ECR repository, authenticate Docker to it, then tag and push:

aws ecr create-repository --repository-name pi-sim --region $AWS_REGION
aws ecr get-login-password --region $AWS_REGION | \
  docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
docker tag pi-sim:latest $ECR_URI:latest
docker push $ECR_URI:latest

3. Create the ECS instance role

Batch launches ECS container instances under the hood, and they need an instance profile to register with ECS. If you've used Batch from the console before, ecsInstanceRole already exists — skip this step. Otherwise:

cat > ec2-trust.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Service": "ec2.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }]
}
EOF
aws iam create-role --role-name ecsInstanceRole \
  --assume-role-policy-document file://ec2-trust.json
aws iam attach-role-policy --role-name ecsInstanceRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role
aws iam create-instance-profile --instance-profile-name ecsInstanceRole
aws iam add-role-to-instance-profile --role-name ecsInstanceRole \
  --instance-profile-name ecsInstanceRole

4. Create the Spot compute environment

This is where the cost savings live. Two choices matter: SPOT_PRICE_CAPACITY_OPTIMIZED picks Spot pools that are both cheap and deep (least likely to be interrupted), and it doesn't require the legacy Spot Fleet IAM role that BEST_FIT needs. Listing several instance families lets Batch dodge capacity-constrained pools.

SUBNETS=$(aws ec2 describe-subnets --filters Name=default-for-az,Values=true \
  --query 'Subnets[].SubnetId' --output json)
SG=$(aws ec2 describe-security-groups --filters Name=group-name,Values=default \
  --query 'SecurityGroups[0].GroupId' --output text)

aws batch create-compute-environment \
  --compute-environment-name sim-spot \
  --type MANAGED --state ENABLED \
  --compute-resources "{
    \"type\": \"SPOT\",
    \"allocationStrategy\": \"SPOT_PRICE_CAPACITY_OPTIMIZED\",
    \"minvCpus\": 0,
    \"maxvCpus\": 1024,
    \"instanceTypes\": [\"c7i\", \"c6i\", \"c5\", \"m7i\", \"m6i\"],
    \"subnets\": $SUBNETS,
    \"securityGroupIds\": [\"$SG\"],
    \"instanceRole\": \"ecsInstanceRole\"
  }"

No --service-role needed — Batch creates and uses the AWSServiceRoleForBatch service-linked role automatically. maxvCpus: 1024 is the fan-out ceiling; raise it to run wider (your EC2 Spot vCPU quota is the real limit — check Service Quotas).

Wait until the environment reports VALID:

aws batch describe-compute-environments --compute-environments sim-spot \
  --query 'computeEnvironments[0].status'

5. Create the job queue

aws batch create-job-queue --job-queue-name sim-queue --priority 1 \
  --compute-environment-order order=1,computeEnvironment=sim-spot

6. Register the job definition

The retry strategy is the critical Spot detail: when EC2 reclaims an instance, the job fails with a status reason matching Host EC2*. This config retries those up to 5 times but exits immediately on genuine application failures, so a bug in your code doesn't burn five attempts.

cat > job-def.json <<EOF
{
  "jobDefinitionName": "pi-sim",
  "type": "container",
  "containerProperties": {
    "image": "$ECR_URI:latest",
    "resourceRequirements": [
      { "type": "VCPU", "value": "1" },
      { "type": "MEMORY", "value": "2048" }
    ]
  },
  "retryStrategy": {
    "attempts": 5,
    "evaluateOnExit": [
      { "onStatusReason": "Host EC2*", "action": "RETRY" },
      { "onReason": "*", "action": "EXIT" }
    ]
  }
}
EOF
aws batch register-job-definition --cli-input-json file://job-def.json

7. Submit the array job

One API call spawns 250 child jobs, each with its own AWS_BATCH_JOB_ARRAY_INDEX:

aws batch submit-job --job-name pi-sweep --job-queue sim-queue \
  --job-definition pi-sim --array-properties size=250

Note the jobId in the response. Array size goes up to 10,000 — combined with a higher maxvCpus, that's your path to thousands of concurrent Spot vCPUs.

Verify it works

Watch the child-status rollup (replace $JOB_ID with the ID from submit-job):

aws batch describe-jobs --jobs $JOB_ID \
  --query 'jobs[0].arrayProperties.statusSummary'

Instances take 2–4 minutes to spin up on the first run. You'll see counts migrate from PENDING through RUNNABLE, RUNNING, and finally:

{
    "STARTING": 0,
    "FAILED": 0,
    "RUNNING": 0,
    "SUCCEEDED": 250,
    "RUNNABLE": 0,
    "SUBMITTED": 0,
    "PENDING": 0
}

Every job's stdout lands in the /aws/batch/job CloudWatch log group:

aws logs tail /aws/batch/job --since 30m

Expected output, one line per shard:

shard=137 samples=10000000 pi_estimate=3.141264 secs=8.3

Troubleshooting

Jobs stuck in RUNNABLE forever. The most common Batch complaint, with three usual causes: the compute environment is INVALID (check describe-compute-environments — the statusReason tells you why), maxvCpus is already consumed by other jobs, or instances launch but never register with ECS because the subnet has no internet route. Default-VPC subnets auto-assign public IPs so ECR and ECS are reachable; private subnets need a NAT gateway or VPC endpoints.

CannotPullContainerError: pull image manifest has been retried 5 time(s): ... not found. The image URI in the job definition doesn't match a pushed image — typically a wrong region in the ECR hostname, a missing :latest push, or a typo'd account ID. Re-run docker push and compare the URI against aws ecr describe-images --repository-name pi-sim.

exec /usr/local/bin/python: exec format error in the job log. You built the image on Apple Silicon without --platform linux/amd64, so it's arm64 and the x86 instances can't execute it. Rebuild with the flag and push again.

Compute environment goes INVALID with CLIENT_ERROR - Access Denied or an instance-profile error. Usually the ecsInstanceRole role exists but the instance profile doesn't — the console creates both together, the CLI doesn't. Run aws iam get-instance-profile --instance-profile-name ecsInstanceRole; if it 404s, redo the last two commands in step 3, then disable/re-enable the environment with aws batch update-compute-environment.

Next steps

Swap the π toy for your real workload: mount input data from S3 (grant the container a jobRoleArn with S3 access), checkpoint anything running longer than ~30 minutes so Spot interruptions cost seconds instead of hours, and keep per-job runtime short — AWS's own Spot guidance favors many small jobs over few large ones. From there, look at fair-share scheduling policies for multi-team queues, jobStateTimeLimitActions to auto-cancel stuck jobs, and AWS Batch on EKS if your infrastructure is Kubernetes-native. The Batch best-practices guide covers the checklist for running at six-figure vCPU scale.