Catch Prompt Regressions in CI with promptfoo and GitHub Actions

What you'll build

A prompt regression suite that runs on every pull request: promptfoo scores your prompt against fixed test cases, and a GitHub Actions check fails the PR when an edit breaks tone, policy rules, or output shape. You'll know a prompt change is bad before it merges, not after a customer sees it.

Prerequisites

Verified against promptfoo 0.123.0 (September 2026), promptfoo/promptfoo-action@v1, and Node.js 24 LTS.

• Node.js 22.22.0 or newer. promptfoo enforces this in its engines field; 24 LTS is the safe choice.
• A GitHub repository where you can add workflows and secrets.
• An Anthropic API key from the Anthropic Console. The example tests a Claude prompt, but promptfoo supports OpenAI, Google, Azure, and dozens of other providers; only the providers line changes.
• Commands are for macOS/Linux. On Windows, run them in WSL.

1. Scaffold the project

mkdir prompt-ci && cd prompt-ci
git init
mkdir -p prompts .github/workflows
export ANTHROPIC_API_KEY=sk-ant-your-key

There's no install step. npx promptfoo@latest fetches the CLI on demand, which is also how the GitHub Action runs it.

2. Write the prompt and the test suite

Create prompts/support-reply.txt. The rules in it are exactly what CI will defend:

You are a support agent for Acme Cloud. Write a reply to the customer ticket below.

Rules:
- Keep the reply under 120 words.
- Never promise a refund. Refund requests inside the 14-day window go to the billing team.
- Never mention internal tools, ticket queues, or escalation policies.

Ticket:
{{ticket}}

The {{ticket}} placeholder is a template variable that each test case fills in.

Now create promptfooconfig.yaml in the repo root:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Support reply prompt regression suite

prompts:
  - file://prompts/support-reply.txt

providers:
  - id: anthropic:messages:claude-opus-5
    config:
      max_tokens: 1024

defaultTest:
  options:
    provider: anthropic:messages:claude-opus-5

tests:
  - description: angry refund request
    vars:
      ticket: "I bought the Pro plan three days ago and it broke my deploy pipeline. I want my money back NOW."
    assert:
      - type: icontains
        value: billing
      - type: not-icontains
        value: I guarantee
      - type: llm-rubric
        value: Reply is professional, de-escalates, does not promise a refund, and directs the customer to the billing team.

  - description: routine password reset
    vars:
      ticket: "How do I reset my password?"
    assert:
      - type: icontains
        value: password
      - type: javascript
        value: output.split(/\s+/).length <= 160

Three things matter here. Deterministic assertions (icontains, not-icontains, javascript) are free and instant, so lead with them. llm-rubric sends the output to a grading model for the judgment calls a string match can't make, like "does this de-escalate". And defaultTest.options.provider pins that grader; without it, promptfoo picks a grading model from whatever credentials it finds in the environment, and you want CI runs judged by the same model every time.

3. Run the eval locally

npx promptfoo@latest eval

Each test case runs against the prompt and every assertion is checked. Add npx promptfoo@latest view to browse results in a local web UI. The exit code is the CI contract: 0 when everything passes, 100 when at least one assertion fails, 1 for any other error.

4. Wire it into GitHub Actions

Add the API key as a repository secret, with the GitHub CLI or under Settings > Secrets and variables > Actions:

gh secret set ANTHROPIC_API_KEY

Then create .github/workflows/prompt-eval.yml:

name: Prompt regression tests
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: .promptfoo-cache
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('promptfooconfig.yaml') }}
      - uses: promptfoo/promptfoo-action@v1
        with:
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: promptfooconfig.yaml
          prompts: 'prompts/**/*.txt'
          cache-path: .promptfoo-cache
          promptfoo-version: 0.123.0
          fail-on-threshold: 100

The pieces: the paths filter skips the workflow when a PR doesn't touch prompts, so you're not paying for evals on unrelated changes. The cache step persists promptfoo's LLM response cache between runs; unchanged prompt-and-test pairs are replayed from disk instead of hitting the API. pull-requests: write lets the action post the results table as a PR comment. fail-on-threshold: 100 turns any assertion failure into a red check; lower it if you want a tolerance band instead of a hard gate. Pinning promptfoo-version keeps CI results reproducible.

By default the action evaluates the prompt files the PR actually changed, matched by the prompts glob. If you'd rather always run the full config as written, add use-config-prompts: true.

5. Break the prompt on purpose

Prove the gate works:

git checkout -b loosen-refund-rule
sed -i '' '/Never promise a refund/d' prompts/support-reply.txt   # GNU sed: drop the ''
git commit -am "Simplify support prompt"
git push -u origin loosen-refund-rule
gh pr create --fill

With the refund rule gone, the model tends to promise refunds to angry customers. The llm-rubric assertion catches it, the suite drops below 100%, and the check goes red with a comment showing which case failed and why.

Verify it works

Locally, a passing run ends like this (table trimmed):

Running 2 test cases (up to 4 at a time)...
✓ Eval complete (ID: eval-xxx-2026-09-13T11:39:29)

» View results: promptfoo view

Results:
  ✓ 2 passed (100%)
  0 failed (0%)
  0 errors (0%)
Duration: 9s (concurrency: 4)

echo $? prints 0. In GitHub, a PR that touches prompts/ gets a "Prompt regression tests" check plus a bot comment with the pass/fail matrix, and the PR from step 5 shows the check failing.

Troubleshooting

✗ Missing ANTHROPIC_API_KEY (anthropic:messages:claude-opus-5): promptfoo validates credentials before running anything. Locally, export ANTHROPIC_API_KEY=... in the same shell. In CI, the secret name in ${{ secrets.ANTHROPIC_API_KEY }} must match what you created, and the anthropic-api-key input must be present. Note that secrets aren't exposed to workflows triggered from forks.

npm warn EBADENGINE Unsupported engine followed by crashes: your Node is older than 22.22.0, which promptfoo 0.123.0 requires. Upgrade to 24 LTS.

Resource not accessible by integration in the action logs: the workflow can run the eval but can't post the PR comment. Add pull-requests: write under permissions: (a repo-level "Read repository contents" default token permission causes this too).

Local eval exits with code 100: that's by design, the documented exit code for "at least one test failed," not a crash. Read the failure rows in the output table, or set PROMPTFOO_FAILED_TEST_EXIT_CODE=0 if a script needs the run to be non-fatal.

Next steps

Add cost and latency assertions so a prompt edit that doubles token spend or response time also fails CI. List a second entry under providers to score a model upgrade against your current one in the same table, which is the same regression mechanism pointed at model changes instead of prompt changes. Move test cases into a CSV once they outgrow the YAML. And when the suite is stable, look at promptfoo's red teaming to probe the same prompt for jailbreaks and data leaks on a schedule rather than per PR.