Benchmark Cursor, Aider, and Claude Code on Your Own Repo

What you'll build

A bash harness that runs Claude Code, Aider, and the Cursor CLI headless against real tasks from your own repo, grades each attempt with a test script, and writes a CSV scoreboard — pass/fail, wall-clock time, and diff size — so your agent choice comes from data, not vibes.

Prerequisites

• macOS or Linux (WSL works). bash and git.
• Verified against: Claude Code 2.1 (native installer), aider 0.86.1, and the current Cursor CLI (agent), all per official docs as of August 2026.
• Python 3.8–3.13 (aider's supported range).
• Accounts: a Claude subscription or an ANTHROPIC_API_KEY; a Cursor account (API key from cursor.com/dashboard/api if you'll run in CI). Aider talks to model APIs directly — we'll point it at the same Anthropic key.
• A repo with a runnable test suite, in a throwaway clone. The harness hard-resets the working tree between runs. Don't point it at your daily checkout.

1. Install the three agents

# Claude Code (native installer)
curl -fsSL https://claude.ai/install.sh | bash
claude   # first run opens browser login; then /exit

# Aider — aider-install puts it in an isolated env in ~/.local/bin
python -m pip install aider-install
aider-install

# Cursor CLI — yes, the binary is literally named `agent`
curl https://cursor.com/install -fsS | bash
agent login   # browser auth; in CI use: export CURSOR_API_KEY=...

# Aider reads this for Anthropic models
export ANTHROPIC_API_KEY=sk-ant-...

Confirm all three respond: claude --version, aider --version, agent --version.

2. Turn real issues into benchmark tasks

Inside your throwaway clone, each task is a folder with a prompt and a grader:

mkdir -p bench/tasks bench/logs
mkdir bench/tasks/issue-142
cat > bench/tasks/issue-142/prompt.md <<'EOF'
Fix the bug where parse_duration("90m") returns 90 seconds instead of
5400. The function lives in src/utils/time.py. Do not add dependencies.
EOF
cat > bench/tasks/issue-142/check.sh <<'EOF'
#!/usr/bin/env bash
python -m pytest tests/test_time.py -q
EOF

The gold standard for task selection: closed issues you've already fixed. On a bench branch, git revert your fix but keep (or cherry-pick) its regression test as check.sh — you get a known-solvable task with an objective grader. Faster alternative: open bugs or small features where you can write the failing test right now. Paste the issue text into prompt.md verbatim; don't add hints you wouldn't give a new teammate. Five to ten tasks is enough to separate the pack.

3. Write the harness

Save as bench/bench.sh:

#!/usr/bin/env bash
# Runs each agent against each task from a clean checkout, grades, logs.
set -u
cd "$(git rev-parse --show-toplevel)"

BASE=$(git rev-parse HEAD)
echo "agent,task,passed,seconds,files_changed,insertions,deletions" > bench/results.csv

run_one() {
  case "$1" in
    claude) claude -p "$2" --model sonnet \
              --dangerously-skip-permissions --max-turns 30 ;;
    aider)  aider --message "$2" --model sonnet --yes-always \
              --no-auto-commits --no-gitignore --no-stream --no-pretty ;;
    cursor) agent -p "$2" --force --output-format text ;;
  esac
}

for task_dir in bench/tasks/*/; do
  task=$(basename "$task_dir")
  prompt=$(<"$task_dir/prompt.md")
  for name in claude aider cursor; do
    git reset --hard -q "$BASE" && git clean -fdq -e bench
    echo "=== $name / $task ==="
    start=$SECONDS
    run_one "$name" "$prompt" > "bench/logs/$name-$task.log" 2>&1
    elapsed=$(( SECONDS - start ))
    if bash "$task_dir/check.sh" > "bench/logs/$name-$task.check.log" 2>&1
      then passed=1; else passed=0
    fi
    stat=$(git diff --shortstat)
    files=$(grep -o '[0-9]* file' <<<"$stat" | grep -o '[0-9]*')
    ins=$(grep -o '[0-9]* insertion' <<<"$stat" | grep -o '[0-9]*')
    del=$(grep -o '[0-9]* deletion' <<<"$stat" | grep -o '[0-9]*')
    echo "$name,$task,$passed,$elapsed,${files:-0},${ins:-0},${del:-0}" \
      >> bench/results.csv
  done
done

git reset --hard -q "$BASE" && git clean -fdq -e bench
column -t -s, bench/results.csv

Three design decisions worth knowing:

• Every run starts from the same commit. git reset --hard plus git clean -fd wipes each agent's changes before the next run; -e bench excludes the harness itself from the wipe.
• Each agent gets its own "just do it" flag, because headless runs can't answer approval prompts: --dangerously-skip-permissions (Claude Code), --yes-always (aider), --force (Cursor). This is exactly why you're in a disposable clone. Aider also gets --no-auto-commits so its edits stay uncommitted and diffable like the others', and --no-gitignore so it doesn't edit .gitignore and pollute the diff stats.
• Models are pinned. All three run Anthropic's Sonnet here (agent --list-models shows Cursor's exact names — add --model to its arm), so you're benchmarking the agent scaffolding, not different models. Drop the --model flags instead if you want to compare each product as shipped, defaults and all.

Diff stats count tracked files only — new files an agent creates won't show — so treat passed as the score and the diff columns as a code-churn tiebreaker.

4. Run the benchmark

chmod +x bench/bench.sh bench/tasks/*/check.sh
./bench/bench.sh

Budget one to five minutes per agent-task pair; a 3×8 matrix is a coffee break. Follow along in another terminal with tail -f bench/logs/*.log.

Verify it works

Before the full matrix, smoke-test with a single trivial task (e.g., "make tests/test_smoke.py pass" with a one-line fix). A healthy run ends with a table like:

agent   task       passed  seconds  files_changed  insertions  deletions
claude  issue-142  1       147      2              38          6
aider   issue-142  1       63       1              12          4
cursor  issue-142  0       201      3              120         41

Every row present, no seconds under ~10 (that usually means the agent errored out instantly — check its log), and git status clean afterward except for bench/.

Troubleshooting

• --dangerously-skip-permissions cannot be used with root/sudo privileges for security reasons — you're running Claude Code as root, typically in a Docker CI image. Create a non-root user, or swap the flag for --permission-mode acceptEdits --allowedTools "Bash".
• aider: command not found or agent: command not found right after installing — both installers drop binaries into ~/.local/bin, which isn't on PATH in fresh shells. export PATH="$HOME/.local/bin:$PATH" or restart the terminal. (Older Cursor CLI installs named the binary cursor-agent; re-run the installer to get agent.)
• bench/tasks/...: No such file or directory on the second task — your bench/ directory got deleted because git clean -fd ran without -e bench. It removes all untracked files; restore the folder and keep the exclude.
• Cursor rows always show passed=0 with zero files changed — print mode won't modify files without --force, and an unauthenticated CLI fails silently into the log. Check agent status, and in CI make sure CURSOR_API_KEY is exported.

Next steps

Agents are nondeterministic, so run each task 3–5 times and report pass rate, not a single coin flip — wrap the inner loop in for trial in 1 2 3. Add a cost column: claude -p --output-format json returns structured results including total cost, and aider prints session cost at the end of each run. Extending the field is one case arm per newcomer — OpenAI's Codex CLI and Google's Gemini CLI slot right in. And when you want to see how your private numbers compare to public ones, SWE-bench is the same idea — real issues, test-based grading — at research scale.