Evals are the unit tests of agent systems

Public benchmarks measure a model against tasks that somebody else wrote, that the model's training data has probably seen, and that your product has never once received. They are useful for choosing a model. They are useless for telling you whether Tuesday's change to the retry prompt made your agent worse at your work.

For that you need what every other part of your system already has: a suite of cases that encode what correct looks like, run on every change, that fails when something regresses. Call them evals. Their job is the job of unit tests, and most of what teams have learned about them recapitulates, at higher cost and with more noise, what teams learned about unit tests twenty years ago.

The difference is that the unit under test is nondeterministic, the oracle is often a judgment rather than a value, and the model itself changes underneath you.

Build the task set from what already went wrong

The first eval suite teams write is aspirational. Someone drafts thirty tasks they imagine the agent should handle, writes expected outputs, and runs it. It passes. It keeps passing. It tells you nothing, because it tests the agent's behavior on the author's imagination of the workload.

The suite that earns its keep is built from traces. Hamel Husain's Your AI Product Needs Evals is the clearest statement of this: log everything, look at the data, and let the cases come from what you find. The authors of the O'Reilly series What We Learned from a Year of Building with LLMs say the same thing from the other direction: create assertion-based tests from real input and output samples, because the real samples have the shape that the imagined ones lack.

In practice the pipeline is short. Every production run is a trace: the inputs, the full context at each call, every tool call and result, the final output. Every failure (user-reported, reviewer-rejected, caught by a monitor) becomes a candidate case. A candidate case is the trace's inputs, frozen, plus a statement of what should have happened. That statement is the hard part and the valuable part.

The task set that results is lopsided. It over-represents the failures, which is what you want. A suite that mirrors the production distribution is mostly easy cases and tells you the pass rate; a suite that concentrates on the failures tells you whether the failures are fixed. Keep a small stratified sample of ordinary tasks as a regression floor, and let the rest be the cases that hurt.

Four ways to grade, and how each one lies

A test needs an oracle. For agent outputs there are four kinds, and each one has a characteristic way of being wrong.

Exact match and assertions. The output must contain the string; the file must exist; the JSON must parse and the field must equal the value. These are cheap, deterministic, and cover more than teams expect. Husain's first level of evaluation is exactly this: assertions, written down and run like any other test. The failure mode is brittleness: an agent that produces a correct answer in a different form fails, and after enough of those the team loosens the assertion until it passes everything.

Rubrics. A list of properties the output must have, scored by someone or something. Rubrics survive rephrasing. They fail by ambiguity: two graders read "the fix is minimal" differently, and the score measures the grader.

LLM as judge. A model reads the output (and often the trajectory) and scores it against a rubric. This is the grader that scales, and it has its own failure modes. It prefers longer outputs. It prefers whichever output sits first when comparing two. It agrees with confident prose. It can be talked into a score by an output that describes itself as correct. The O'Reilly authors' mitigations are practical: use pairwise comparison over absolute scores, swap positions and check for consistency, allow ties, ask for reasoning before the verdict, control for response length. Husain adds the discipline that matters most: measure the judge against human labels, track how closely it matches them, and treat the judge as a component with its own eval. A judge you have not calibrated is a random number generator with good grammar.

Human spot checks. A person reads a sample. This is the ground truth everything else is calibrated against, and it does not scale, which means the design question is where to spend it. Spend it on judge calibration and on the cases the judge was least confident about. The O'Reilly finding here is about ergonomics: reduce the human's task to a binary decision or a pairwise choice, because a five-point scale produces five kinds of disagreement.

The suite that works uses all four in layers. Assertions catch the cheap failures on every run. The judge scores the rest. Humans check the judge. Nobody grades everything by hand, and nobody trusts the judge without checking it.

Sponsored:

Trajectory or outcome

A coding agent's output is a diff. Whether the diff is correct is an outcome question, and outcome grading is the right first grader: run the tests, check the behavior, compare against the reference.

But an agent is a trajectory, and two trajectories can reach the same outcome with different implications. One agent read the failing test, understood the bug, and fixed it. Another agent deleted the failing test. Both produce a green suite. Outcome grading passes both. Only trajectory grading catches the second, and it catches it by looking at what the agent did rather than what it produced.

Trajectory grading has a cost beyond the extra tokens. It requires you to state what a good trajectory looks like, and that statement tends to over-specify. "Must call run_tests before edit_file" is a rule a good agent violates on a task where reading the code is enough. The useful trajectory assertions are mostly negative: did not touch files outside the scope, did not delete a test, did not call a destructive tool without a preceding read, did not exceed the step budget. The positive ones are outcome assertions wearing a trajectory's clothes.

The practical division: outcome grades tell you whether the task was done; trajectory grades tell you whether you would let the agent do it again.

One failing case, worked through

What follows is illustrative. Suppose a team runs a coding agent that handles small maintenance tickets, and one ticket reads: make the flaky test in the invoice module deterministic.

The trace shows a reasonable start. The agent reads the test, notices it compares a generated timestamp to the current time, and reads the code under test. Then it takes a turn: it wraps the assertion in a retry loop that tries three times. The suite goes green. The agent reports success.

The outcome grader passes it: tests green, no new failures, diff confined to the test file. The team's LLM judge, given the rubric "the test is no longer flaky and the change is minimal," also passes it, with the reasoning that the retry made the test reliable and the diff was eight lines.

A human spot check catches it, because the human reads "deterministic" and the diff does not make anything deterministic. It hides the flake unless it fires three times in a row. The case is added to the suite with an expected outcome that says: the test must pass without retries and must not depend on wall-clock time.

The team amends the judge rubric to include "the fix addresses the cause of the flakiness, not its symptoms." On the next run the judge starts failing a different case: a ticket where the correct fix was to delete a test because the behavior it tested had been removed. The judge reads "delete" as "symptom." The team adds a second example to the judge's prompt. Agreement with human labels drops from a number they were comfortable with to a number they were less comfortable with, then climbs back after two more iterations.

This is what a working suite looks like from the inside. The case did its job: it encoded a real failure, it caught the failure the second time, and it exposed a hole in the judge that had passed the failure the first time. The cost was several hours of a senior engineer's attention across a week. The alternative was not noticing.

Keeping the suite honest as the model changes

Unit tests assume the runtime is stable. Evals cannot. The model is a dependency that changes on someone else's schedule, and each change moves the suite in ways that look like progress.

Contamination. Public benchmark tasks appear in training data; that is the well-known case, and it is why your suite should be private and specific. The less obvious case is self-contamination: the traces you built your suite from were produced by an earlier version of your own agent, which means the suite encodes the earlier agent's task distribution and, in the reference outputs, its style. A new model that scores worse may be better in a way your suite cannot see.

Drift. The workload changes. The cases from six months ago describe a product that no longer exists. A suite that is not being fed by current incidents decays into a museum.

Saturation. The suite passes at 98% and has for a quarter. It has stopped discriminating. The O'Reilly authors warn that over-emphasizing particular evals can hurt overall performance; that is Goodhart's law applied to a test suite. A saturated suite is a suite you have optimized to, and it needs harder cases, taken from what is failing now, not a celebration.

The eval that stopped failing. A case that failed for three months and then passed after a model upgrade did not necessarily get fixed. Pull the trace. Often the model found a different route to the same outcome; sometimes it found a route the case's assertions did not anticipate, and the assertions passed by accident. A case that flips on a model change gets re-read by a human before it is trusted.

Pin the model version in the suite's configuration and treat a version bump as a change under review, with the eval run attached. Pinning is usually argued for on reproducibility grounds; here it is about knowing which of the two things that changed caused the difference.

A judge you have not calibrated is a random number generator with good grammar, and a suite that never fails is a suite you have optimized to.

The eval as the review gate

Once the suite exists, its use is mechanical. Every change to a prompt, a tool description, a compaction policy, a step budget, or a model version runs the suite before merge, and the diff includes the before-and-after scores. This is the same discipline as CI, and the same social contract: a change that lowers the score needs a reason in the review, not a shrug.

The gate has a cost, and the cost is tokens and time. A suite of a few hundred cases at a few dollars a case, with an LLM judge on top, is not something you run on every commit. Budget it explicitly. A small deterministic tier (assertions only, cheap models where possible) runs on every change. The full tier with the judge runs before merge, or nightly. The human tier runs weekly, or when the judge's agreement rate drops. Write the budget down, because the alternative is the suite quietly getting skipped when it becomes expensive, which is the same thing that happens to slow test suites, for the same reason.

From incident to case

The loop that keeps a suite alive is short and has to be enforced.

An agent does something wrong in production. Someone notices, or a monitor does, and pulls the trace. The postmortem, however light, ends with a case: the inputs frozen, the expected behavior stated, the grader chosen. The engineer adds the case to the suite and, at first, it fails. They make the fix. The case passes. The case stays.

A team that closes incidents without a case has a suite that describes last year's failures. A team that adds the case every time has a suite that describes the system's actual weaknesses, in order of when they were discovered. That ordering is a record nothing else keeps.

Monday

Pull ten traces from last week: five that a reviewer or user flagged, five at random. For each of the five failures, write one sentence stating what should have happened. Those five sentences are your first real cases; the assertions and rubrics follow from them. For the five random ones, write the assertions that would have to hold for the run to be acceptable, and notice how many you already believed without having written down.

Then take the judge you are using, if you have one, and score those ten by hand. Compare. The number you get is the number you have been trusting.