Prompt Evaluation

In DSPy you do not write the prompt itself. You list what goes in and what should come out. DSPy calls this list a signature, and it writes the actual prompt for you. This keeps what you want apart from how you ask. The “how you ask” part is what an optimiser can change.

class MultipleChoiceQuestion(dspy.Signature):
    """Write one multiple-choice exam question about the passage."""

    passage: str = dspy.InputField()
    question: str = dspy.OutputField()
    options: list[str] = dspy.OutputField()
    answer_index: int = dspy.OutputField()

There are 24 hand-written passages. Each comes with a hand-written short-answer question and a hand-written multiple-choice question. These are the reference questions: examples of what good looks like. Twelve passages are for training, which means improving the prompt. The other twelve are held back: kept aside and used only for testing. Each principle appears once in each half.

The optimisers only ever see the training half. Every score on this page comes from the held-back half. So a prompt cannot score well by memorising its test.

This is the most important step. The metric is the scoring rule: a small piece of code that gives one output a score. Here it is a checklist anyone can read. No second AI acts as a judge:

def short_answer_checks(passage, pred):
    q, a = str(pred.question).strip(), str(pred.answer).strip()
    return {
        "is_a_question": q.endswith("?") and len(q.split()) >= 5,
        "stands_alone": stands_alone(q),
        "answer_is_short": 0 < len(a.split()) <= 12,
        "answer_grounded": overlap(a, passage) >= 0.6,
        "answer_not_leaked": overlap(a, q) < 0.6,
    }

Before trusting it, the metric was run on the 48 hand-written reference questions, which should all pass. They did not, at first: see the lessons.

Run the plain prompt on the held-back passages and score every output. This first score is the baseline: the starting point that every change is compared with. Then run the same test again. If the score moves between identical runs, that movement is noise (random variation). Differences between prompts that are smaller than the noise mean nothing.

evaluate = dspy.Evaluate(devset=devset, metric=mcq_metric, num_threads=4)
evaluate(dspy.Predict(MultipleChoiceQuestion))

Here are four ways to improve the prompt. All four are scored the same way. The last three use a DSPy optimiser.

optimiser = dspy.MIPROv2(metric=mcq_metric, auto="light", seed=7)
better = optimiser.compile(dspy.Predict(MultipleChoiceQuestion), trainset=trainset)
better.save("results/mipro_mcq_program.json")

The saved file is the improved prompt: its instructions and its chosen examples. It is a plain text (JSON) file, so you can keep each version and review the changes.

Plus 3 worked example(s).

Plus 3 worked example(s).

Plus 3 worked example(s).

Plus 3 worked example(s).

The first metric only checked that outputs had the right format. The plain prompt scored 83 and 96 out of 100. That looked good, but the questions were poor. Answers ran to 50 words. Questions said “according to the passage”, but students would never see the passage. A high score from a weak metric gives false comfort. Decide what good means, in terms you can check, before you try to improve anything.

Run the metric on answers you know are good. Also read a few outputs yourself. Here, one reference question failed because it was an instruction (“Give one way…”), not a question. A strict word match also marked a correct model answer as not grounded. The answer said “building” where the passage said “builds”. Then a stricter length check was added, and 15 of the 24 hand-written multiple-choice questions failed it. The author had made the longest option the correct one. That is exactly the flaw the check was added to catch in the model. People make the mistakes they look for in machines.

BootstrapFewShot showed the model three short, correct answers, and it still wrote long ones. One line of instruction, “The answer must be 12 words or fewer”, fixed almost every answer. Small models copy the format of examples more easily than qualities nobody states, such as being short.

MIPROv2 asks the model to suggest better instructions, based on the data and the program. But it never shows the model the metric. The suggestions added drama (“a high-stakes certification exam”). They also narrowed the topic to three principles the model happened to see. None mentioned answer length. When MIPROv2 started from the hand-written rules instead, it gave the highest short-answer score. That was a little above the rules alone. On multiple choice it tied. In this experiment the person’s rules did most of the work. The optimiser added examples and polish.

The hand-written rules told the model to keep all options the same length, and to write the wrong options in its own words. After that, fewer correct answers stood out. But the model also reworded the correct answer, so it drifted away from the passage. When MIPROv2 then rewrote those instructions, the correct answers matched the passage again. But the correct option was once again often the longest. Check every rule after every change, not only the one you were fixing.

Every prompt put the correct answer at B or C far more often than chance, and almost never at A. Each question is written separately, so no prompt can balance a set. Two lines of code can: shuffle the options and update the index. Use prompts for things that need judgement. Use code for fixed rules.

order = random.sample(range(4), 4)
options, answer_index = [options[i] for i in order], order.index(answer_index)

The same prompt, run five times on the same passages, moved by up to about 3 points. That is the noise. With only twelve test passages, one question is worth several points. Differences smaller than the noise are ties, however good the chart looks. More test cases shrink the noise. More tests always beat cleverer tests.

The metric checks format, and whether answers are grounded in the passage. It cannot tell if a question tests understanding or just memory. It cannot tell if a question is fair to every student, or even correct. That still needs a subject expert. Use the score to compare prompt versions, and to spot when a change makes things worse. Use people to approve the questions.

Every run writes the model, DSPy version, seed, data split, prompt, outputs and scores to results/. That record is what lets someone else check the claim, and what you rerun when the model, the brief or the data changes. Without that record, a prompt is just an experiment. With it, the prompt is a governed part of your system.

“Model not found”: the download in step 2 did not finish, or you downloaded a different file. For the smaller Q4_0 file, give its path: MODEL=/path/to/gemma-4-E2B-it-Q4_0.gguf ./scripts/start_server.sh.

Port 8090 already in use: start on another port and tell the scripts where it is: PORT=8091 ./scripts/start_server.sh, then in the second terminal export LLM_API_BASE=http://localhost:8091/v1 before uv run.

hf or llama-server not found: open a new terminal so the install is on your PATH. On Linux, set LLAMA_SERVER=~/src/llama.cpp/build/bin/llama-server if you built it elsewhere.

No graphics card: everything still works on the CPU, only more slowly. The three-minute figure is for a desktop GPU; CPU times were not measured.

Scores differ from this page: expected, by a few points. See lesson 07.