Build CloudWatch Dashboards, Metric Filters, and Alarms for Lambda

What you'll build

A production-grade monitoring stack for a Lambda workload: a metric filter that turns structured error logs into a custom CloudWatch metric, two child alarms (application errors and p95 latency) rolled into one composite alarm that pages via SNS, and a single dashboard showing traffic, latency, alarm state, and the actual error logs.

Prerequisites

• An AWS account with permissions for Lambda, IAM, CloudWatch, CloudWatch Logs, and SNS.
• AWS CLI version 2, configured with credentials and a default region. Commands verified against the v2 command reference, July 2026.
• Lambda runtime python3.13 (everything here also works on python3.14).
• bash or zsh on macOS/Linux. On Windows, use WSL.

Set two variables every step reuses:

export AWS_REGION=us-east-1
ACCT=$(aws sts get-caller-identity --query Account --output text)

1. Deploy an instrumented Lambda function

The demo function simulates a payment processor. The key detail: a declined payment is a handled failure — the function returns normally, so Lambda's built-in Errors metric never sees it. That's exactly the gap metric filters close.

# lambda_function.py
import logging
import random
import time

logger = logging.getLogger()

def handler(event, context):
    time.sleep(random.uniform(0.05, 0.3))
    if event.get("mode") == "fail":
        logger.error("payment declined", extra={"errorType": "CardDeclined", "amount": 129.0})
        return {"status": "declined"}
    logger.info("payment captured", extra={"amount": 129.0})
    return {"status": "ok"}

Create the execution role and the function. LogFormat=JSON makes every logging call land in CloudWatch as structured JSON with timestamp, level, message, and requestId keys — which is what makes the filter pattern in step 2 trivial.

cat > trust.json <<'EOF'
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "lambda.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}
EOF

ROLE_ARN=$(aws iam create-role --role-name pay-fn-role \
  --assume-role-policy-document file://trust.json \
  --query Role.Arn --output text)

aws iam attach-role-policy --role-name pay-fn-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

zip function.zip lambda_function.py

aws lambda create-function \
  --function-name pay-fn \
  --runtime python3.13 \
  --handler lambda_function.handler \
  --zip-file fileb://function.zip \
  --role "$ROLE_ARN" \
  --timeout 10 \
  --logging-config LogFormat=JSON,ApplicationLogLevel=INFO,SystemLogLevel=WARN

If create-function complains the role can't be assumed, wait ten seconds and rerun — that's IAM propagation, covered in Troubleshooting.

2. Extract a custom metric from the logs

Invoke the function once so the log group exists (metric filters can't attach to log groups that don't exist yet), then cap retention:

aws lambda invoke --function-name pay-fn \
  --cli-binary-format raw-in-base64-out \
  --payload '{"mode":"ok"}' /dev/null

aws logs put-retention-policy --log-group-name /aws/lambda/pay-fn --retention-in-days 14

Now the metric filter. The pattern { $.level = "ERROR" } matches any JSON log event whose level property equals ERROR. Each match publishes 1 to a custom metric; defaultValue=0 publishes a zero when logs arrive that don't match, which keeps the metric alive during healthy traffic:

aws logs put-metric-filter \
  --log-group-name /aws/lambda/pay-fn \
  --filter-name payment-errors \
  --filter-pattern '{ $.level = "ERROR" }' \
  --metric-transformations \
    metricName=PaymentErrors,metricNamespace=PayService,metricValue=1,defaultValue=0,unit=Count

3. Create the SNS topic and child alarms

TOPIC_ARN=$(aws sns create-topic --name pay-alerts --query TopicArn --output text)
aws sns subscribe --topic-arn "$TOPIC_ARN" \
  --protocol email --notification-endpoint you@example.com

Check your inbox and click Confirm subscription — unconfirmed subscriptions silently drop notifications.

Create two child alarms with no actions — paging is the composite's job, so a blip in one signal alone never wakes anyone. Both use M-of-N evaluation (2 breaching datapoints out of 3 one-minute periods) to ignore single-minute spikes, and notBreaching for missing data since zero traffic means zero matched log events, not an outage:

aws cloudwatch put-metric-alarm \
  --alarm-name pay-fn-app-errors \
  --namespace PayService --metric-name PaymentErrors \
  --statistic Sum --period 60 \
  --threshold 5 --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --treat-missing-data notBreaching

aws cloudwatch put-metric-alarm \
  --alarm-name pay-fn-latency-p95 \
  --namespace AWS/Lambda --metric-name Duration \
  --dimensions Name=FunctionName,Value=pay-fn \
  --extended-statistic p95 --period 60 \
  --threshold 3000 --comparison-operator GreaterThanThreshold \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --treat-missing-data notBreaching

4. Combine them into a composite alarm

The composite goes to ALARM when either child does, and it's the only thing wired to SNS:

aws cloudwatch put-composite-alarm \
  --alarm-name pay-fn-user-impact \
  --alarm-rule 'ALARM("pay-fn-app-errors") OR ALARM("pay-fn-latency-p95")' \
  --alarm-actions "$TOPIC_ARN"

Rules support AND, OR, NOT, and parentheses — a common production pattern is AND NOT ALARM("deploy-in-progress") to suppress pages during deploys.

5. Build the dashboard

One JSON body, four widgets: traffic vs. failures, latency percentiles, live alarm status, and a Logs Insights table of the most recent errors:

cat > dashboard.json <<EOF
{
  "widgets": [
    {
      "type": "metric", "x": 0, "y": 0, "width": 12, "height": 6,
      "properties": {
        "title": "Invocations vs handled failures",
        "region": "$AWS_REGION", "period": 60, "stat": "Sum",
        "metrics": [
          ["AWS/Lambda", "Invocations", "FunctionName", "pay-fn"],
          ["PayService", "PaymentErrors", {"color": "#d62728"}]
        ]
      }
    },
    {
      "type": "metric", "x": 12, "y": 0, "width": 12, "height": 6,
      "properties": {
        "title": "Duration (ms)",
        "region": "$AWS_REGION", "period": 60,
        "metrics": [
          ["AWS/Lambda", "Duration", "FunctionName", "pay-fn", {"stat": "p50"}],
          ["AWS/Lambda", "Duration", "FunctionName", "pay-fn", {"stat": "p95"}]
        ]
      }
    },
    {
      "type": "alarm", "x": 0, "y": 6, "width": 12, "height": 6,
      "properties": {
        "title": "Paging status",
        "alarms": [
          "arn:aws:cloudwatch:$AWS_REGION:$ACCT:alarm:pay-fn-app-errors",
          "arn:aws:cloudwatch:$AWS_REGION:$ACCT:alarm:pay-fn-latency-p95",
          "arn:aws:cloudwatch:$AWS_REGION:$ACCT:alarm:pay-fn-user-impact"
        ]
      }
    },
    {
      "type": "log", "x": 12, "y": 6, "width": 12, "height": 6,
      "properties": {
        "title": "Latest errors",
        "region": "$AWS_REGION", "view": "table",
        "query": "SOURCE '/aws/lambda/pay-fn' | filter level = \"ERROR\" | fields @timestamp, message, errorType, requestId | sort @timestamp desc | limit 20"
      }
    }
  ]
}
EOF

aws cloudwatch put-dashboard --dashboard-name pay-service \
  --dashboard-body file://dashboard.json

Expected response — an empty validation array means the body parsed cleanly:

{
    "DashboardValidationMessages": []
}

Verify it works

Simulate a three-minute incident (about 30 declined payments per minute):

for i in $(seq 1 90); do
  aws lambda invoke --function-name pay-fn \
    --cli-binary-format raw-in-base64-out \
    --payload '{"mode":"fail"}' /dev/null > /dev/null
  sleep 2
done

After two to three minutes of failures, check alarm state. Note --alarm-types — without it, describe-alarms returns metric alarms only and your composite is invisible:

aws cloudwatch describe-alarms --alarm-types MetricAlarm CompositeAlarm \
  --query '{metric: MetricAlarms[].[AlarmName,StateValue], composite: CompositeAlarms[].[AlarmName,StateValue]}'

Expected output:

{
    "metric": [
        ["pay-fn-app-errors", "ALARM"],
        ["pay-fn-latency-p95", "OK"]
    ],
    "composite": [
        ["pay-fn-user-impact", "ALARM"]
    ]
}

You should also have an email from pay-alerts, and the dashboard (CloudWatch console → Dashboards → pay-service) shows the red PaymentErrors line spiking, two alarm tiles red, and the declined payments in the log table.

Troubleshooting

An error occurred (InvalidParameterValueException) when calling the CreateFunction operation: The role defined for the function cannot be assumed by Lambda. IAM roles take a few seconds to propagate after creation. Wait ~10 seconds and rerun create-function unchanged.

Invalid base64: "{"mode": "fail"}" AWS CLI v2 expects base64-encoded --payload by default. Add --cli-binary-format raw-in-base64-out (as the commands above do), or make it permanent with aws configure set cli-binary-format raw-in-base64-out.

An error occurred (ResourceNotFoundException) when calling the PutMetricFilter operation: The specified log group does not exist. /aws/lambda/pay-fn is only created on first invocation. Invoke the function once, then retry the metric filter.

Alarm stuck in INSUFFICIENT_DATA. Metric filters only emit datapoints when log events arrive — no traffic, no data. Make sure --treat-missing-data notBreaching is set and generate a few invocations. Also confirm the metric exists: aws cloudwatch list-metrics --namespace PayService should list PaymentErrors.

Next steps

• Front the function with API Gateway and repeat the pattern on its access logs: JSON access logging plus a filter like { $.status = 5* } gives you a 5XX metric per route, and AWS/ApiGateway 5XXError/Latency widgets slot straight into the dashboard body.
• Alarm on rates, not counts — CloudWatch metric math (errors / invocations * 100) via put-metric-alarm --metrics scales with traffic where a fixed threshold won't.
• Try anomaly detection alarms for latency, which learn a band instead of a static threshold.
• Move it into IaC — this whole stack is four resources in CloudFormation or Terraform; put-dashboard bodies paste directly into an AWS::CloudWatch::Dashboard resource.
• Read AWS's alarm best practices for recommended thresholds per service.