Optimize LLM Prompts Automatically with DSPy Instead of Hand-Tuning

What you'll build

A support-ticket classifier where you never hand-write the prompt: you declare the task as a DSPy signature, score it against a small labeled dataset, and let the MIPROv2 optimizer compile an instruction + few-shot demos that measurably beat your baseline.

Prerequisites

• Python 3.10+ (any OS; commands below assume macOS/Linux — on Windows, activate the venv with .venv\Scripts\activate)
• DSPy 3.3.0 (verified against the docs for the August 2026 release)
• An Anthropic API key from the Anthropic Console. DSPy talks to providers through LiteLLM, which is installed automatically as a dependency — no separate SDK needed.

Expect the optimization run to make a few hundred model calls. DSPy caches LM responses on disk by default, so re-runs are nearly free.

1. Install DSPy and configure the model

python -m venv .venv && source .venv/bin/activate
pip install "dspy==3.3.0"
export ANTHROPIC_API_KEY="sk-ant-..."

Create optimize.py and start with the LM setup:

import dspy

lm = dspy.LM("anthropic/claude-opus-5", max_tokens=16000)
dspy.configure(lm=lm)

Two deliberate choices here: we don't pass temperature — DSPy 3.3 omits it unless you set it, and Anthropic's current models (Opus 4.7 and later) reject sampling parameters with a 400. And max_tokens=16000 gives headroom because Claude Opus 5 reasons before answering by default, and max_tokens caps reasoning plus response together.

2. Declare the task as a signature, not a prompt

In DSPy you specify what goes in and out; the framework renders the actual prompt. Typed output fields with Literal constrain the model to your label set:

from typing import Literal

Label = Literal["billing", "bug", "how_to", "feature_request"]

class ClassifyTicket(dspy.Signature):
    """Classify a customer support ticket."""
    ticket: str = dspy.InputField()
    label: Label = dspy.OutputField()

classify = dspy.Predict(ClassifyTicket)

That docstring is the entire "prompt" you write by hand. MIPROv2 will rewrite it for you in step 4.

3. Build the scored dataset and a metric

Optimization needs labeled examples and a metric that scores predictions. Twenty-four short tickets are enough to see real movement:

raw = [
    ("I was charged twice this month", "billing"),
    ("Refund hasn't shown up after 10 days", "billing"),
    ("Why did my plan price go up?", "billing"),
    ("Invoice PDF shows the wrong company name", "billing"),
    ("Card declined but you still billed me", "billing"),
    ("Need a receipt for my last payment", "billing"),
    ("App crashes when I upload a PNG", "bug"),
    ("Export button does nothing on Safari", "bug"),
    ("Getting a 500 error on login since today", "bug"),
    ("Dark mode resets after every restart", "bug"),
    ("Search returns results from deleted projects", "bug"),
    ("Notifications arrive twice on Android", "bug"),
    ("How do I invite a teammate?", "how_to"),
    ("Where do I change my email address?", "how_to"),
    ("Can I export my data to CSV?", "how_to"),
    ("How do I set up SSO with Okta?", "how_to"),
    ("What's the keyboard shortcut for search?", "how_to"),
    ("How do I cancel my subscription?", "how_to"),
    ("Please add a calendar view", "feature_request"),
    ("Would love webhook support for Zapier", "feature_request"),
    ("Any plans for an offline mode?", "feature_request"),
    ("A bulk-edit option would save me hours", "feature_request"),
    ("Please support markdown in comments", "feature_request"),
    ("It would be great to pin favorite projects", "feature_request"),
]

data = [dspy.Example(ticket=t, label=l).with_inputs("ticket") for t, l in raw]
train, dev = data[::2], data[1::2]   # 12 train, 12 dev, balanced by label

def accuracy(example, pred, trace=None):
    return example.label == pred.label

with_inputs("ticket") tells DSPy which fields are inputs; everything else is treated as the gold label.

4. Score the baseline, then compile with MIPROv2

evaluate = dspy.Evaluate(devset=dev, metric=accuracy,
                         num_threads=8, display_progress=True)

baseline = evaluate(classify)
print(f"Baseline: {baseline.score}")

optimizer = dspy.MIPROv2(metric=accuracy, auto="light", num_threads=8)
optimized = optimizer.compile(classify, trainset=train, valset=dev,
                              minibatch=False)

print(f"Optimized: {evaluate(optimized).score}")
optimized.save("ticket_classifier.json")

Run it with python optimize.py. MIPROv2 bootstraps candidate few-shot demos from train, proposes rewritten instructions, and searches over combinations, scoring each candidate program on valset with your metric — automated prompt engineering against numbers instead of vibes. auto="light" bounds the budget; minibatch=False matters because the default minibatch size (35) exceeds our tiny valset and would error. The saved JSON contains the winning instruction and demos.

5. Verify it works

The run takes a few minutes. You should see evaluation progress bars and MIPROv2's trial logging, ending with something like:

Average Metric: 8.00 / 12 (66.7%): 100%|██████████| 12/12
Baseline: 66.67
...
===== Trial 7 / 10 =====
Best full score so far! Score: 91.67
...
Optimized: 91.67

Exact numbers vary between runs, but optimized should beat baseline. Confirm the compiled program reloads cleanly by appending:

loaded = dspy.Predict(ClassifyTicket)
loaded.load("ticket_classifier.json")
print(loaded(ticket="You billed my old credit card again"))
# Prediction(label='billing')

dspy.inspect_history(n=1)

inspect_history prints the last actual prompt sent to Claude — you'll see an instruction you never wrote, plus selected few-shot examples. That's the compiled artifact.

Troubleshooting

litellm.AuthenticationError: AnthropicException - authentication_error ... invalid x-api-key — the key isn't visible to the script. Export ANTHROPIC_API_KEY in the same shell you run python from (or pass api_key=... to dspy.LM), and check for a stale key from an old shell profile.

ValueError: Minibatch size cannot exceed the size of the valset — you removed minibatch=False (or shrank the dataset). Either keep minibatch=False, pass a smaller minibatch_size to compile(), or use a valset with 35+ examples.

litellm.BadRequestError: AnthropicException - ... invalid_request_error mentioning temperature — something is setting sampling parameters, which Anthropic's newest models reject. Remove any temperature/top_p arguments from your dspy.LM(...) call; on older DSPy versions (pre-3.3, where temperature defaulted to 0.0) upgrade with pip install -U dspy.

AdapterParseError or truncated/missing label field — the response hit the token cap before the answer, usually because reasoning consumed the budget. Raise max_tokens in dspy.LM(...).

Next steps

MIPROv2 is the workhorse, but DSPy 3.x's GEPA optimizer often does better on harder tasks by using a strong reflection model to evolve instructions from natural-language feedback — your metric can return dspy.Prediction(score=..., feedback="...") instead of a bare number. Swap dspy.Predict for dspy.ChainOfThought or dspy.ReAct and recompile — the optimizer works on any module unchanged. For real projects, grow the dataset (50–300 examples), hold out a test split the optimizer never sees, and check the compiled JSON into version control so prompt changes get code review like everything else. The MIPROv2 API reference documents every knob used here, including the medium and heavy budgets for larger datasets.