Build a Multi-Region AWS DR Plan and Prove Your RTO With a Drill

What you'll build

A working disaster recovery pipeline on AWS: hourly backups of a DynamoDB table in us-east-1, automatic copies to a vault in us-west-2 via AWS Backup, and a scripted failover drill that restores the cross-region copy and passes or fails against a 30-minute RTO. The drill is a rerunnable shell script, so it can become a monthly calendar item instead of a wiki page nobody tests.

Prerequisites

• AWS CLI v2, verified against 2.35.x (aws --version)
• An AWS account with permissions to manage IAM roles, AWS Backup, and DynamoDB
• Two regions that support cross-region backup. This tutorial uses us-east-1 (primary) and us-west-2 (DR)
• bash or zsh; commands tested on macOS 15 and work unchanged on Linux
• Cost: a few cents for backup storage and the restored table. Cleanup commands are in Next steps

Set the variables every later command uses:

export AWS_PAGER=""
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export PRIMARY=us-east-1
export DR=us-west-2

1. Pick targets and create a test table

Write your two numbers down before touching infrastructure. RPO (recovery point objective) is how much data you can lose; with hourly backups plus copy lag, worst case here is about 90 minutes. RTO (recovery time objective) is how long until you're serving again; we'll target 30 minutes and make the drill prove it.

Create a table to protect and seed one item you'll look for after failover:

aws dynamodb create-table --table-name orders \
  --attribute-definitions AttributeName=pk,AttributeType=S \
  --key-schema AttributeName=pk,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST --region $PRIMARY

aws dynamodb put-item --table-name orders \
  --item '{"pk": {"S": "order-1001"}, "status": {"S": "shipped"}}' \
  --region $PRIMARY

2. Enable advanced DynamoDB backup

Cross-region copy of DynamoDB backups only works with AWS Backup's advanced features turned on (accounts that first used AWS Backup after November 2021 have them on by default). Check:

aws backup describe-region-settings --region $PRIMARY \
  --query '{mgmt: ResourceTypeManagementPreference.DynamoDB, optin: ResourceTypeOptInPreference.DynamoDB}'

If either value is false, enable both:

aws backup update-region-settings --region $PRIMARY \
  --resource-type-opt-in-preference DynamoDB=true \
  --resource-type-management-preference DynamoDB=true

3. Create vaults in both regions and a service role

One vault per region; the DR vault must exist before the plan can copy into it:

aws backup create-backup-vault --backup-vault-name dr-primary --region $PRIMARY
aws backup create-backup-vault --backup-vault-name dr-secondary --region $DR

AWS Backup assumes an IAM role to do the work. Create one with the two AWS managed policies:

cat > trust.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "backup.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

aws iam create-role --role-name dr-backup-role \
  --assume-role-policy-document file://trust.json
aws iam attach-role-policy --role-name dr-backup-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForBackup
aws iam attach-role-policy --role-name dr-backup-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForRestores
export ROLE_ARN=arn:aws:iam::$ACCOUNT_ID:role/dr-backup-role

4. Create a backup plan with a cross-region copy

The CopyActions block is what makes this a DR plan rather than a backup plan. Every scheduled backup lands in dr-primary, then AWS Backup copies it to dr-secondary in us-west-2, re-encrypting with the destination vault's key:

cat > backup-plan.json <<EOF
{
  "BackupPlanName": "dr-plan",
  "Rules": [{
    "RuleName": "hourly-with-dr-copy",
    "TargetBackupVaultName": "dr-primary",
    "ScheduleExpression": "cron(0 * ? * * *)",
    "StartWindowMinutes": 60,
    "CompletionWindowMinutes": 180,
    "Lifecycle": {"DeleteAfterDays": 35},
    "CopyActions": [{
      "DestinationBackupVaultArn": "arn:aws:backup:$DR:$ACCOUNT_ID:backup-vault:dr-secondary",
      "Lifecycle": {"DeleteAfterDays": 35}
    }]
  }]
}
EOF

export PLAN_ID=$(aws backup create-backup-plan --backup-plan file://backup-plan.json \
  --region $PRIMARY --query BackupPlanId --output text)

aws backup create-backup-selection --backup-plan-id $PLAN_ID --region $PRIMARY \
  --backup-selection "{\"SelectionName\": \"orders\", \"IamRoleArn\": \"$ROLE_ARN\", \"Resources\": [\"arn:aws:dynamodb:$PRIMARY:$ACCOUNT_ID:table/orders\"]}"

5. Take the first backup and copy it over

The hourly rule will handle both jobs automatically from now on, but an on-demand backup doesn't run the plan's copy actions, so for the first pass trigger both yourself:

export JOB_ID=$(aws backup start-backup-job --backup-vault-name dr-primary \
  --resource-arn arn:aws:dynamodb:$PRIMARY:$ACCOUNT_ID:table/orders \
  --iam-role-arn $ROLE_ARN --region $PRIMARY --query BackupJobId --output text)

aws backup describe-backup-job --backup-job-id $JOB_ID --region $PRIMARY \
  --query '{State: State, RecoveryPoint: RecoveryPointArn}'

Rerun that last command until State is COMPLETED (a small table takes a few minutes), then kick off the copy:

export RP_ARN=$(aws backup describe-backup-job --backup-job-id $JOB_ID \
  --region $PRIMARY --query RecoveryPointArn --output text)

export COPY_ID=$(aws backup start-copy-job --recovery-point-arn $RP_ARN \
  --source-backup-vault-name dr-primary \
  --destination-backup-vault-arn arn:aws:backup:$DR:$ACCOUNT_ID:backup-vault:dr-secondary \
  --iam-role-arn $ROLE_ARN --region $PRIMARY --query CopyJobId --output text)

aws backup describe-copy-job --copy-job-id $COPY_ID --region $PRIMARY \
  --query CopyJob.State

Wait for "COMPLETED". You now have a recovery point sitting in us-west-2 that survives the loss of us-east-1.

6. Script the failover drill

The drill restores the newest recovery point in the DR vault to a fresh table and times it. EncryptionType=Default is required metadata here: a restore in a different region than the source table must state its encryption, and Default uses an AWS-owned key.

cat > drill.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
DR=us-west-2
RTO_TARGET_MIN=30
TABLE="orders-drill-$(date +%Y%m%d%H%M)"

RP=$(aws backup list-recovery-points-by-backup-vault \
  --backup-vault-name dr-secondary --region "$DR" \
  --query 'sort_by(RecoveryPoints, &CreationDate)[-1].RecoveryPointArn' --output text)
echo "Restoring $RP as $TABLE"

START=$(date +%s)
JOB=$(aws backup start-restore-job --recovery-point-arn "$RP" \
  --iam-role-arn "$ROLE_ARN" --resource-type DynamoDB --region "$DR" \
  --metadata TargetTableName="$TABLE",EncryptionType=Default \
  --query RestoreJobId --output text)

while :; do
  S=$(aws backup describe-restore-job --restore-job-id "$JOB" \
    --region "$DR" --query Status --output text)
  [ "$S" = COMPLETED ] && break
  if [ "$S" = FAILED ] || [ "$S" = ABORTED ]; then echo "Restore $S"; exit 1; fi
  sleep 15
done

ELAPSED=$(( ($(date +%s) - START) / 60 ))
echo "Restore completed in ${ELAPSED} min (RTO target: ${RTO_TARGET_MIN} min)"
if [ "$ELAPSED" -le "$RTO_TARGET_MIN" ]; then echo "DRILL PASSED"; else echo "DRILL FAILED: RTO missed"; fi
EOF
chmod +x drill.sh && ./drill.sh

ROLE_ARN comes from your shell environment, so run the script from the same session (or hardcode it for the scheduled version).

Verify it works

The drill prints its own verdict. Expect something like:

Restoring arn:aws:backup:us-west-2:123456789012:recovery-point:1e2f3a4b-... as orders-drill-202609141410
Restore completed in 11 min (RTO target: 30 min)
DRILL PASSED

A near-empty table restores in roughly 5 to 15 minutes. Now confirm the data actually made the round trip, using the table name the drill printed:

aws dynamodb get-item --table-name orders-drill-202609141410 \
  --key '{"pk": {"S": "order-1001"}}' --region $DR

{
    "Item": {
        "pk": {"S": "order-1001"},
        "status": {"S": "shipped"}
    }
}

If the seeded item comes back from a table that never existed in us-west-2, your DR path works end to end. Note what the drill proves and what it doesn't: it times data recovery, not application cutover. Your real RTO adds DNS or client failover on top, so keep the restore comfortably under target.

Troubleshooting

• An error occurred (AccessDeniedException) when calling the StartBackupJob operation: Insufficient privileges to perform this action. The role is missing a managed policy or doesn't trust backup.amazonaws.com. Recheck step 3. IAM is also eventually consistent, so a role used seconds after creation can throw this once; wait a minute and retry.
• Error parsing parameter '--backup-plan': Invalid JSON The shell mangled your quoting. Pass the plan as file://backup-plan.json as shown instead of inlining JSON on the command line.
• The copy job fails or the recovery point ARN starts with arn:aws:dynamodb instead of arn:aws:backup. That backup was taken without advanced DynamoDB features, and those recovery points can't be copied cross-region. Enable the features (step 2), take a new backup, and copy that one. Enabling doesn't upgrade old recovery points.
• The restore job goes FAILED with an InvalidParameterValueException mentioning encryption. Cross-region DynamoDB restores must specify an encryption key. Keep EncryptionType=Default in the metadata, or pass EncryptionType=KMS,KmsMasterKeyArn=<key arn> with a key that exists in the DR region.

Next steps

Automate the drill: AWS Backup restore testing plans (aws backup create-restore-testing-plan) run scheduled restores and record whether they succeeded, which turns this script into a managed control. Extend coverage by adding RDS, EBS, and EFS ARNs to the backup selection; only the restore metadata changes per service. For the cutover half of RTO, look at Route 53 health checks with failover routing. And harden the vault with AWS Backup Vault Lock so a compromised account can't delete your recovery points.

When you're done, delete the drill tables and the orders table, remove the recovery points from both vaults, then delete the vaults, the backup plan, and the role. Recovery points bill until deleted.