Evals for LLM apps: a starter kit you can run today

Most LLM features are tested the same way: you try five inputs, read the outputs, and they look fine. Then someone changes one line of the prompt to fix a complaint, and nobody knows what else moved. An eval replaces "looks fine" with a number you can compare: a fixed set of inputs, answers you agree with, and code that scores the model against them.

✦ In short

An eval is a golden set (inputs plus the answers you want), checks written in code, and a habit of reading the failures before changing anything.
I built one in about 200 lines of standard-library Python and ran it against qwen2.5:3b through Ollama on an 8 GB M2 MacBook. The task: pull vendor, date, amount and currency out of 30 receipt sentences.
The first prompt scored 0 out of 30 under strict checks. Reading the failures showed the model mostly understood the receipts; the prompt never said what format I wanted. Writing that spec into the prompt took it to 30 out of 30.
A perfect score on 30 cases is weaker evidence than it looks. The statistics still allow a failure rate of up to 9.5%, the prompt was written by someone who had read those 30 cases, and on 10 fresh sentences it scored 9 out of 10.
For open-ended outputs, an LLM judge can help, but it has known biases. Check it against your own labels before you trust it.
In CI, gate on which cases changed, not only the average.

The loop

The harness is the least important part. What makes evals work is the loop around it:

01Collect real inputs
02Label the answers
03Check with code
04Read the failures
05Change one thing
06Re-run the same set

Step 4 is where the learning happens. Everything else exists so that step is cheap.

Re-running exactly the same set is what makes a before and after comparable.

Everything below runs locally. The model is qwen2.5:3b (Alibaba's Qwen2.5 family, released in September 2024) served by Ollama, with temperature 0 and a fixed seed. Ollama's API docs describe exactly that combination for reproducible outputs. The code uses only Python's standard library.

A task small enough to finish

The example is field extraction: turn one sentence about a payment into JSON with four keys. It's a common job for an LLM inside an app (expense trackers, invoice inboxes, bookkeeping) and it has right answers, which makes it a good first eval.

The golden set is 30 invented sentences. Every vendor is made up. One JSON object per line:

golden.jsonl

{"id": 1, "text": "Paid ₹450 to Asha's Bakery on 12 March 2025.", "expected": {"vendor": "Asha's Bakery", "date": "2025-03-12", "amount": 450, "currency": "INR"}, "tags": []}
{"id": 4, "text": "Lunch at Kaveri Meals on 03/04/2025, total ₹680.", "expected": {"vendor": "Kaveri Meals", "date": "2025-04-03", "amount": 680, "currency": "INR"}, "tags": ["dd/mm"]}
{"id": 12, "text": "Booked a table at Saffron House for 20 July 2025; we pay on arrival.", "expected": {"vendor": "Saffron House", "date": "2025-07-20", "amount": null, "currency": null}, "tags": ["missing field"]}
{"id": 30, "text": "IronCore Fitness membership, ₹2,500 a month, paid for 3 months on 1 June 2025.", "expected": {"vendor": "IronCore Fitness", "date": "2025-06-01", "amount": 7500, "currency": "INR"}, "tags": ["arithmetic"]}

Twelve of the 30 are plain. The other 18 carry tags for the things I expected to be hard: day-first numeric dates, amounts that need arithmetic (a tip, VAT, three months at a monthly price), "₹1.2 lakh", Indian digit grouping like ₹2,35,000, a European €12.400,00, a missing field, two dates in one sentence, and a location stuck to a shop name.

Labelling forced decisions I hadn't made yet. Is 03/04/2025 the 3rd of April or the 4th of March? (Day first: these are Indian receipts.) Is the amount the subtotal or the total? (The total, including tax and tip.) Does the vendor include "Indiranagar"? (No.) Those decisions are the spec. If you can't label a case, the model can't get it right either.

Checks written in code

Each output gets three checks, from cheap to strict:

Is it JSON? Parse it. A ```json fence around it is allowed, nothing else.
Is it the right shape? Exactly the four keys, with each value either null or the right type: an ISO date string, a number, a three-letter currency code.
Is each field right? Exact match against the label, after trivial normalisation (case, extra spaces, a trailing full stop). Amounts match to within 0.005.

A case passes only if all four fields are right. I also report each field on its own, because "70% overall" hides whether the date or the amount is the problem.

harness.py

import argparse
import json
import re
import urllib.request
 
from prompts import PROMPTS
 
FIELDS = ("vendor", "date", "amount", "currency")
 
 
def ask(prompt, text, temperature=0.0, seed=42):
    body = {
        "model": "qwen2.5:3b",
        "messages": [{"role": "user", "content": prompt.replace("{text}", text)}],
        "stream": False,
        "options": {"temperature": temperature, "seed": seed},
    }
    req = urllib.request.Request(
        "http://localhost:11434/api/chat",
        data=json.dumps(body).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=300) as resp:
        return json.load(resp)["message"]["content"]
 
 
def parse(raw):
    """Check 1: is it JSON? Allows a ```json fence around it, nothing else."""
    raw = re.sub(r"^```(?:json)?\s*|\s*```$", "", raw.strip())
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError:
        return None
    return obj if isinstance(obj, dict) else None
 
 
RULES = {
    "vendor": lambda v: isinstance(v, str) and v.strip() != "",
    "date": lambda v: isinstance(v, str) and re.fullmatch(r"\d{4}-\d{2}-\d{2}", v),
    "amount": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool),
    "currency": lambda v: isinstance(v, str) and re.fullmatch(r"[A-Z]{3}", v),
}
 
 
def schema_errors(obj):
    """Check 2: exactly the right keys, and each value null or the right shape."""
    errors = [] if set(obj) == set(FIELDS) else [f"keys={sorted(obj)}"]
    for field, ok in RULES.items():
        value = obj.get(field)
        if value is not None and not ok(value):
            errors.append(f"{field}={value!r}")
    return errors
 
 
def same(field, got, want):
    """Check 3: exact match per field, after trivial normalisation."""
    if field == "amount" and isinstance(got, (int, float)) and want is not None:
        return abs(got - want) < 0.005
    if isinstance(got, str) and isinstance(want, str):
        return " ".join(got.casefold().split()).rstrip(".") == " ".join(
            want.casefold().split()
        ).rstrip(".")
    return got == want
 
 
def grade(case, raw):
    obj = parse(raw)
    row = {"id": case["id"], "tags": case["tags"], "raw": raw, "valid_json": obj is not None}
    if obj is None:
        return row | {"schema_ok": False, "fields": dict.fromkeys(FIELDS, False), "pass": False}
    errors = schema_errors(obj)
    fields = {f: same(f, obj.get(f), case["expected"][f]) for f in FIELDS}
    return row | {
        "got": obj,
        "schema_errors": errors,
        "schema_ok": not errors,
        "fields": fields,
        "pass": not errors and all(fields.values()),
    }
 
 
def report(rows):
    n = len(rows)
    lines = [("valid JSON", sum(r["valid_json"] for r in rows)),
             ("schema ok", sum(r["schema_ok"] for r in rows))]
    lines += [(f, sum(r["fields"][f] for r in rows)) for f in FIELDS]
    lines += [("all 4 fields", sum(r["pass"] for r in rows))]
    for name, k in lines:
        print(f"{name:13s}{k:3d}/{n}  {k / n:6.1%}")
 
 
if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("prompt", choices=PROMPTS)
    ap.add_argument("--data", default="golden.jsonl")
    ap.add_argument("--temperature", type=float, default=0.0)
    ap.add_argument("--seed", type=int, default=42)
    ap.add_argument("--out", required=True)
    args = ap.parse_args()
 
    cases = [json.loads(line) for line in open(args.data)]
    rows = []
    for case in cases:
        raw = ask(PROMPTS[args.prompt], case["text"], args.temperature, args.seed)
        rows.append(grade(case, raw))
    with open(args.out, "w") as f:
        f.writelines(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)
    print(f"prompt {args.prompt}, temperature {args.temperature}, seed {args.seed}")
    report(rows)

Every raw output is saved next to its grade. You will want to read them.

Run 1: zero out of thirty

The first prompt is the one most people write first:

prompts.py

    "v1": """Extract the vendor, date, amount and currency from this receipt text.
Reply with JSON only, using the keys vendor, date, amount, currency.
 
{text}""",

python3 harness.py v1 --out results/v1.jsonl

prompt v1, temperature 0.0, seed 42
valid JSON    30/30  100.0%
schema ok      0/30    0.0%
vendor        29/30   96.7%
date           2/30    6.7%
amount         1/30    3.3%
currency      15/30   50.0%
all 4 fields   0/30    0.0%

Every output parsed as JSON, and not one passed. That's the point of per-field scores: the vendor was right 29 times out of 30, so the model clearly read the sentences. Something else was going on.

Read the failures before you change anything

A short script groups the failures by field and by tag, then prints each one next to its label:

analyse.py

import json
import sys
from collections import Counter
 
FIELDS = ("vendor", "date", "amount", "currency")
data = sys.argv[2] if len(sys.argv) > 2 else "golden.jsonl"
cases = {c["id"]: c for c in map(json.loads, open(data))}
rows = [json.loads(line) for line in open(sys.argv[1])]
fails = [r for r in rows if not r["pass"]]
print(f"{len(fails)} of {len(rows)} cases failed")
 
wrong = Counter(f for r in fails for f in FIELDS if not r["fields"][f])
print("wrong field:", ", ".join(f"{f} {k}" for f, k in wrong.most_common()))
 
seen = Counter(t for c in cases.values() for t in c["tags"] or ["plain"])
failed = Counter(t for r in fails for t in r["tags"] or ["plain"])
for tag, k in seen.most_common():
    print(f"  {tag:17s}{failed[tag]}/{k} failed")
 
for r in fails:
    print(f"\n#{r['id']} {cases[r['id']]['text']}")
    for f in FIELDS:
        if not r["fields"][f]:
            got = r.get("got", {}).get(f, "(no JSON)")
            print(f"   {f}: want {cases[r['id']]['expected'][f]!r}, got {got!r}")

The first lines of its output:

30 of 30 cases failed
wrong field: amount 29, date 28, currency 15, vendor 1
  plain            12/12 failed
  dd/mm            6/6 failed
  arithmetic       4/4 failed
  missing field    2/2 failed

The tags don't help: plain cases failed as often as hard ones. Reading the cases does. The commonest failure looked like this:

#1 Paid ₹450 to Asha's Bakery on 12 March 2025.
   date: want '2025-03-12', got '12 March 2025'
   amount: want 450, got '450'
   currency: want 'INR', got '₹'

The model copied the date as written, returned the amount as a string, and used the symbol instead of a code. None of that is wrong in spirit. I never said I wanted ISO dates, numbers or ISO 4217 codes. Going through all 30, I counted 25 that had the right information in the wrong form. The model even did the arithmetic on the tip and the VAT cases, and returned the answers as strings.

The other five were real misunderstandings, and these are the ones worth keeping:

#7 Paid the Sharma Tutorials fee of ₹1.2 lakh on 1 April 2025.
   amount: want 120000, got '1.2'
   currency: want 'INR', got 'lakh rupees'
 
#21 Invoice from Atlas Logistics, total due €12.400,00, date 05.03.2025.
   amount: want 12400, got '12.400'
 
#27 Urban Threads order (₹1,999) paid on 10 Aug 2025, delivered 12 Aug 2025.
   date: want '2025-08-10', got '12 Aug 2025'
 
#28 Asha's Bakery, Indiranagar: ₹260 on 2 Sept 2025.
   vendor: want "Asha's Bakery", got "Asha's Bakery, Indiranagar"
 
#30 IronCore Fitness membership, ₹2,500 a month, paid for 3 months on 1 June 2025.
   amount: want 7500, got '2500'

(Trimmed to the fields that matter. Each case also had the format failures above.)

So the failures fall into two groups: format (the spec was never written down) and meaning (lakh, European number format, which date counts, what belongs in a name, quantity times price). That grouping says what to change. A bigger model wouldn't reliably fix the first group, because no model can guess a format nobody asked for.

Change one thing, and re-run the same set

The second prompt writes the spec down, with one worked example. The example uses a different vendor and different numbers from anything in the golden set, so the set isn't leaking into the prompt.

prompts.py

    "v2": """Extract the payment from the text. Reply with one JSON object and nothing else.
 
Keys:
- vendor: the business that was paid, as written, without a location.
- date: the payment or invoice date as YYYY-MM-DD. Numeric dates are day first,
  so 03/04/2025 means 3 April 2025.
- amount: the final total paid, as a plain number. Add tax and tips, multiply
  quantities, and expand "lakh" (100000) and "k" (1000). Never return a subtotal.
- currency: an ISO 4217 code. ₹, Rs and rupees are INR; $ is USD; € is EUR; £ is GBP.
- Use null for anything the text doesn't say.
 
Example
Text: Paid Rs 1,20,500 to Lakeview Interiors, Whitefield on 04/09/2024 (subtotal Rs 1,02,119 + GST).
JSON: {"vendor": "Lakeview Interiors", "date": "2024-09-04", "amount": 120500, "currency": "INR"}
 
Text: {text}
JSON:""",

prompt v2, temperature 0.0, seed 42
valid JSON    30/30  100.0%
schema ok     30/30  100.0%
vendor        30/30  100.0%
date          30/30  100.0%
amount        30/30  100.0%
currency      30/30  100.0%
all 4 fields  30/30  100.0%

Thirty out of thirty, from a 3B model running on a laptop, including all five of the meaning failures: 120000 for the lakh, 12400 for the European invoice, the payment date rather than the delivery date, the vendor without the neighbourhood, and 7500 for three months of membership.

first prompt, strict checks: 0/30; first prompt, strict checks
of those failures were format, not meaning: 25; of those failures were format, not meaning
after writing the spec into the prompt: 30/30; after writing the spec into the prompt
failure rate 30 passes can't rule out: 9.5%; failure rate 30 passes can't rule out

Is 30 out of 30 real?

A perfect score should make you more suspicious, not less. Three questions are worth asking.

Did the prompt overfit the set?

The v2 rules were written by someone who had read every one of those 30 cases, and each rule maps to a case in the set. That is how prompt work goes, and it means the score on those 30 is optimistic. The honest check is cases the prompt wasn't written against, so I wrote 10 new sentences in the same style and ran them once.

python3 harness.py v2 --data heldout.jsonl --out results/v2_heldout.jsonl
python3 analyse.py results/v2_heldout.jsonl heldout.jsonl

prompt v2, temperature 0.0, seed 42
valid JSON    10/10  100.0%
schema ok     10/10  100.0%
vendor        10/10  100.0%
date           9/10   90.0%
amount        10/10  100.0%
currency      10/10  100.0%
all 4 fields   9/10   90.0%
1 of 10 cases failed
wrong field: date 1
  dd/mm            1/3 failed
  arithmetic       0/2 failed
  plain            0/1 failed
  lakh             0/1 failed
  missing field    0/1 failed
  several amounts  0/1 failed
  location in name 0/1 failed
  two dates        1/1 failed
 
#110 Patel Furnishings invoice for ₹48,500, dated 01-06-2025, delivered 10-06-2025.
   date: want '2025-06-01', got '2025-06-10'

Nine out of ten, and the one failure is the two-dates problem again: it took the delivery date. The golden set had a single case with two dates, the prompt fixed that one, and a slightly different sentence broke it. That's what overfitting to an eval set looks like at small scale. The fix is to move case 110 into the golden set and keep writing fresh cases, ideally from real inputs, every time the prompt changes.

How much does a run move on its own?

Ollama's docs say a fixed seed makes the model return the same text for the same prompt, and every run above used temperature 0 and seed 42, once per prompt. The variation you can't avoid comes from sampling. Ollama's default temperature is 0.8, and plenty of apps never change it. To see what that does, I ran the five hardest cases three more times at temperature 0.8 with seeds 1, 2 and 3:

variance.py

import json
 
from harness import ask, grade
from prompts import PROMPTS
 
HARD = {7, 21, 23, 27, 30}  # lakh, European format, VAT, two dates, months x price
cases = [c for c in map(json.loads, open("golden.jsonl")) if c["id"] in HARD]
 
for seed in (1, 2, 3):  # temperature 0.8 is Ollama's default
    rows = [grade(c, ask(PROMPTS["v2"], c["text"], temperature=0.8, seed=seed)) for c in cases]
    failed = [r["id"] for r in rows if not r["pass"]]
    print(f"seed {seed}: {sum(r['pass'] for r in rows)}/{len(rows)} pass, failed: {failed or 'none'}")
    for r in rows:
        if not r["pass"]:
            print(f"   #{r['id']} got {r.get('got', r['raw'])}")

seed 1: 5/5 pass, failed: none
seed 2: 4/5 pass, failed: [23]
   #23 got {'vendor': 'Maple & Co.', 'date': '2025-06-10', 'amount': 12600, 'currency': 'GBP'}
seed 3: 4/5 pass, failed: [23]
   #23 got {'vendor': 'Maple & Co.', 'date': '2025-06-10', 'amount': 12600, 'currency': 'GBP'}

Same prompt, same cases, and the VAT invoice (£1,050 plus 20%) came back as 1260 at temperature 0 and with seed 1, and as 12600 with seeds 2 and 3: ten times too much. A case that passes once at temperature 0 isn't necessarily solid. It may be one sample that happened to land on the right answer.

Three runs of five cases is a small experiment, kept small to be kind to an 8 GB laptop, but the lesson holds. If the feature runs with sampling switched on, evaluate it with sampling switched on, run each case several times, and report the pass rate per case. A case that passes two times out of three is a different thing from one that passes every time.

How noisy are 30 examples?

Even with perfectly repeatable runs, 30 cases is a small sample of everything users will type. The bootstrap is a simple way to see how small. Resample the 30 cases with replacement, re-score, do that 10,000 times, and read off the middle 95% of the scores:

bootstrap.py

import json
import random
import sys
 
FIELDS = ("vendor", "date", "amount", "currency")
 
 
def ci(values, draws=10_000, seed=0):
    """95% percentile bootstrap: resample the cases with replacement, re-score."""
    rng = random.Random(seed)
    n = len(values)
    means = sorted(sum(values[rng.randrange(n)] for _ in range(n)) / n for _ in range(draws))
    return means[int(0.025 * draws)], means[int(0.975 * draws)]
 
 
for path in sys.argv[1:]:
    rows = [json.loads(line) for line in open(path)]
    print(path)
    metrics = {f: [r["fields"][f] for r in rows] for f in FIELDS}
    metrics["all 4 fields"] = [r["pass"] for r in rows]
    for name, values in metrics.items():
        lo, hi = ci(values)
        print(f"  {name:13s}{sum(values):3d}/{len(values)}  95% CI {lo:6.1%} to {hi:6.1%}")
    if all(metrics["all 4 fields"]):
        n = len(rows)
        print(f"  no failures: true failure rate could still be up to {1 - 0.05 ** (1 / n):.1%}")

results/v1.jsonl
  vendor        29/30  95% CI  90.0% to 100.0%
  date           2/30  95% CI   0.0% to  16.7%
  amount         1/30  95% CI   0.0% to  10.0%
  currency      15/30  95% CI  33.3% to  66.7%
  all 4 fields   0/30  95% CI   0.0% to   0.0%
results/v2_t0_run1.jsonl
  vendor        30/30  95% CI 100.0% to 100.0%
  date          30/30  95% CI 100.0% to 100.0%
  amount        30/30  95% CI 100.0% to 100.0%
  currency      30/30  95% CI 100.0% to 100.0%
  all 4 fields  30/30  95% CI 100.0% to 100.0%
  no failures: true failure rate could still be up to 9.5%

Look at v1's currency field: 15 out of 30, with an interval from 33% to 67%. With 30 cases, a "50%" could easily be a 35% or a 65%. Two prompts that score 50% and 60% on this set are not meaningfully different.

The v2 lines show where the bootstrap breaks. If every case passes, every resample passes too, and the interval collapses to 100% to 100%. That doesn't mean certainty. The last line uses the exact binomial bound instead: if 30 out of 30 passed, a true failure rate as high as 9.5% is still consistent with what we saw, at 95% confidence. The quick version is the "rule of three", from a 1983 paper by Hanley and Lippman-Hand: with zero failures in n cases, the upper bound is about 3/n.

The practical rule: the width of the interval shrinks with the square root of the number of cases, so halving it takes four times as many. Thirty cases tell you about big changes, like 0 to 30. To tell 85% from 90%, you need a few hundred.

When there's no exact answer: LLM-as-judge

Field extraction is the easy case, because the checks are code. Summaries, support replies and chat answers don't have one right string. The common answer is to ask a stronger model to grade the output, and it does work, but it has failure modes that are now well documented.

The paper most people cite is Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023, NeurIPS 2023 Datasets and Benchmarks track). It found that GPT-4 as a judge agreed with human preferences over 80% of the time, about the same rate at which humans agree with each other. It also named the biases:

Position bias. The judge prefers an answer because of where it appears. When the paper swapped the order of two answers, GPT-4 gave the same verdict 65.0% of the time, GPT-3.5 46.2%, and Claude-v1 23.8%. An earlier paper, Large Language Models are not Fair Evaluators (Wang et al., 2023), showed that with ChatGPT as the judge, Vicuna-13B could beat ChatGPT on 66 of 80 questions just by changing the order of the answers.
Verbosity bias. The judge prefers longer answers. In the paper's "repetitive list" test, where an answer was padded with reworded copies of its own points, GPT-3.5 and Claude-v1 preferred the padded answer 91.3% of the time. GPT-4 did so 8.7% of the time.
Self-enhancement bias. A judge may favour answers written by itself. Zheng et al. saw signs of it but couldn't confirm it statistically with their data. A 2024 paper by Panickssery, Bowman and Feng found that models like GPT-4 and Llama 2 can recognise their own outputs, and that the stronger the self-recognition, the stronger the self-preference.

None of this means "don't use a judge". It means a judge is a model too, and needs its own eval.

✦ Calibrating a judge against your own labels

Label first. Grade 30 to 50 real outputs yourself, pass or fail, with a sentence on why. Those labels are the judge's golden set.
Ask for pass or fail, not a score out of 10. Binary verdicts are easier to compare with your labels and easier to act on.
Measure the two errors separately. Of the outputs you failed, how many did the judge fail? Of the ones you passed, how many did it pass? If 90% of outputs are good, a judge that always says "pass" gets 90% agreement and catches nothing.
Swap the order in pairwise comparisons, and count a win only when the verdict holds both ways. Zheng et al. treat a disagreement as a tie.
Hold some labels back. Tune the judge prompt on some of them and test it once on the rest, for the same reason as the held-out receipts above.
Prefer code where you can. A JSON check, a regex or "does the reply mention the refund window?" is cheaper, faster and never biased towards long answers.

A judge that agrees with you on held-out labels is a useful tool. One that has never been compared with a human is a second opinion from someone you haven't met.

Evals as a regression gate in CI

Once the set and the checks exist, putting them in CI is a small step. The question is what to gate on. An average threshold alone is weak: with 30 cases, one regression and one new fix cancel out and the average doesn't move. With temperature 0 and a fixed seed, a local run is meant to be repeatable, so I'd gate on which cases changed:

gate.py

import json
import sys
 
FLOOR = 0.80  # never ship below this, whatever the baseline says
 
 
def passes(path):
    return {r["id"]: r["pass"] for r in map(json.loads, open(path))}
 
 
baseline, current = passes(sys.argv[1]), passes(sys.argv[2])
score = sum(current.values()) / len(current)
broke = [i for i in baseline if baseline[i] and not current[i]]
fixed = [i for i in baseline if not baseline[i] and current[i]]
 
print(f"score {score:.1%} (baseline {sum(baseline.values()) / len(baseline):.1%})")
print(f"newly failing: {broke or 'none'}")
print(f"newly passing: {fixed or 'none'}")
if broke or score < FLOOR:
    sys.exit("eval gate: FAIL")
print("eval gate: pass")

With the v2 run saved as the baseline, re-checking v2 passes. Pointing it at the v1 results, as if someone had reverted the prompt, fails with a non-zero exit code and names every case that broke:

score 100.0% (baseline 100.0%)
newly failing: none
newly passing: none
eval gate: pass

score 0.0% (baseline 100.0%)
newly failing: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30]
newly passing: none
eval gate: FAIL

In a CI job that's two commands, run whenever a pull request touches the prompts, the golden set or the model settings:

python3 harness.py v2 --out results/current.jsonl
python3 gate.py results/baseline.jsonl results/current.jsonl

A few practical points:

The runner needs the model. For a local model that means a self-hosted runner with Ollama on it. For a hosted API it means a key and a budget. Thirty short calls cost very little; three hundred on every push adds up, so run on the pull requests that touch prompts, not on every commit.
Changing the baseline is a decision. When a change is meant to move cases, commit the new results file in the same pull request, so a reviewer sees which cases moved.
Every production bug becomes a case. When a user hits a failure, add it to the golden set with the right label before fixing it. That's how the set gets hard enough to be useful, and how 30 cases become 300.

What I'd take away

The most useful moment in this whole exercise was reading 30 failures and seeing that 25 of them were my fault. The model had read the receipts correctly; nobody had said what was wanted. If I had jumped from 0% to "try a bigger model", I would have paid for more parameters to fix a missing paragraph.

The second lesson is to distrust a clean score. 30 out of 30 sounds finished. It's 30 cases I chose, a prompt I wrote while looking at them, and an interval that still allows roughly one failure in ten. Ten fresh sentences found a miss straight away.

If I were adding evals to an LLM feature tomorrow, I'd start with 20 to 30 real inputs, label them by hand, write the checks as code, and read every failure before changing anything. The harness here is about 200 lines of standard-library Python. It doesn't need a framework to be useful; it needs someone to read what it prints.