Evals Are the Only Regression Suite an AI System Will Ever Give You

You can't write assertEquals against a system that doesn't return the same answer twice. Every team building on an LLM discovers this in week one — and most respond by not writing tests at all.

I've spent two decades on systems where the regression suite was the contract. If CI was green, you deployed; if a change broke behavior, a red build caught it before a customer did. Then I started shipping LLM-backed features and watched that contract dissolve. Temperature zero doesn't restore it: with batched inference and floating-point accumulation, the same prompt can produce different tokens on Tuesday than it did on Friday. The deterministic toolbox — fixtures, snapshots, exact assertions — tests the 20% of your system that was never going to break.

The common conclusion is that AI systems can't be tested, so you ship and watch dashboards. I think that reading is backwards. AI systems can be tested — the discipline exists, it's called evals, and the teams treating evals with the same rigor as CI are the ones whose agents survive contact with production. In the final post of our agentic-loop series I argued the loop is the easy 20% and the guardrails are the other 80%. Evals are the half of that 80% almost everyone skips.

A model upgrade is a dependency bump nobody reviews

Start with why regressions happen at all in a system you didn't redeploy. Your AI feature's behavior is a function of at least four inputs that change independently:

• The model. Providers retrain and re-route behind stable aliases. The model your prompt was tuned against in March is not the model serving it in July, and no changelog will tell you what shifted.
• The prompt. Every prompt edit is a deploy. Teams that require review for a one-line config change will let a 40-line system prompt change ship on a Slack thumbs-up.
• The context. If you run retrieval, your RAG index mutates daily as documents get added. Same prompt, same model, different grounding — different answer.
• The tools. Rename a parameter in a tool schema and the agent that called it correctly for months starts guessing.

Each of these is a deploy that no test gates. In classical software we'd call that continuous delivery with no pipeline. That's the gap the eval suite fills: a fixed set of cases, run on every change, that tells you whether behavior moved.

The golden dataset is the spec you never wrote

An eval suite starts with a golden dataset: real inputs with labeled expected behavior. The operative word is real. The failure I keep meeting in audits is a golden set built from the demo script — twenty tidy cases the team invented, all passing, while the feature degrades on the traffic users actually send.

Build it from production traces instead. Capture every interaction from day one — inputs, retrieved context, tool calls, outputs — and then sit down and label a hundred of them. It's unglamorous work, and it's also where the product gets defined: the moment someone has to write down what the correct behavior is for a messy real case, you surface every disagreement the team has been deferring. The golden dataset is the spec you never wrote, extracted one labeled case at a time.

Then layer the assertions by cost. Deterministic checks first: the output parses as JSON, the right tool was called, the refund amount is under the policy cap, no email address appears in the response. These are cheap, fast and unambiguous — a surprising fraction of what "good behavior" means reduces to them. Semantic checks second, for the judgments code can't make: was the answer grounded in the retrieved context, was the tone right, did it decline the out-of-scope request. That's where the judge comes in.

An LLM judge is a reviewer you also have to review

Using a strong model to grade your system's outputs is standard practice now, and the foundational result holds up: the MT-Bench paper (Zheng et al., NeurIPS 2023) found GPT-4's agreement with human raters above 80% — the same rate humans reach with each other. A judge model is the only scalable way to grade semantics.

It is also a component with its own failure modes, documented in that same paper: position bias (preferring whichever answer it saw first), verbosity bias (rewarding longer answers regardless of quality), self-enhancement bias (favoring text that resembles its own output). A judge is a flaky reviewer — useful, mostly right, systematically wrong in specific directions.

So you review the reviewer. Calibrate the judge against a slice of human labels — twenty or thirty cases is enough to measure agreement — and treat a drop in that agreement as a red build. Pin the judge's model version the way you pin any dependency, and re-calibrate when you bump it, because a judge upgrade shifts your metrics with zero change in the system being measured. I've watched a team celebrate a five-point quality jump that was entirely the judge getting more lenient.

Four ways an eval lies to you

A green eval run is evidence, not proof. The strongest objection to everything above is that eval suites become theater — a fixed set of cases passing forever while the product rots. That's a real failure mode, and it has four specific mechanisms:

1. Golden-set overfitting. Tune your prompt against the same hundred cases for six months and you've trained on your test set — Goodhart's law with extra steps. Keep a hold-out slice the prompt authors never see.
2. Judge drift. The grader moved, the system didn't. Pinning and calibration, as above.
3. Leakage. Golden cases quietly get pasted into the prompt as few-shot examples, and the eval now measures memorization. Track provenance in both directions.
4. Staleness. Your traffic changed and the golden set didn't. The eval keeps certifying performance on last quarter's problem.

The countermeasure to all four is the same: keep the dataset alive. Rotate fresh production traces in every month, retire cases that no longer represent traffic, and let the failures you find in production become next month's regression cases — the exact discipline of adding a test for every bug.

Wire it like CI, not like a research project

The mechanics, concretely:

• Gate changes. Prompt edits, tool-schema changes and retrieval config land through pull requests that trigger the eval suite. A material score drop blocks the merge, exactly like a failing unit test.
• Run nightly against the live alias. Your pinned model is what you tested; the provider's alias is what you serve. A nightly run against both is how you find out the ground moved before your users do.
• Sample production. Route 1–5% of live traffic through the judge, asynchronously, and alert on week-over-week drift. This is the check the golden set can't do — it catches the regression on traffic you didn't predict.
• Budget it honestly. Grading 200 cases with a frontier model costs single-digit dollars per run. One production incident retro costs more than a year of nightly evals.

None of this requires exotic tooling — open-source harnesses like promptfoo or OpenAI Evals cover the mechanics — because the hard part was never the runner. It's the labeled dataset and the habit.

What I'd do this quarter if I were your CTO

1. Turn on trace capture today. Inputs, context, tool calls, outputs, model version. Every other step depends on this data existing.
2. Label 100 golden cases from real traces. Product and engineering in the same room. Budget two days; the arguments it forces are the deliverable.
3. Put the suite in CI. Prompt changes gate on it. If your prompts aren't in version control, that's step zero.
4. Calibrate your judge monthly. Twenty human-labeled cases, measure agreement, pin the judge version.
5. Sample 2% of production through the judge with a drift alarm. When it fires, the failing traces become new golden cases.

A system that never behaves the same way twice can still be held to a standard — you just have to write the standard down and check it on every change, which is all a regression suite ever was. The teams shipping reliable agents aren't the ones with better models. They're the ones who rebuilt their testing discipline for a probabilistic component instead of abandoning it.

---

If you're standing up this discipline and need engineers who have run it in production — golden sets, judge calibration, eval-gated CI — that's what our LLM engineers do.