The demo proved the model could do the thing once. Production asks a harder question, and it asks it every time something changes: is the model still doing the thing after someone edited the prompt on Tuesday? You can’t answer that by re-reading a single output, and you can’t answer it by asking the model how it thinks it’s doing. You answer it with evals.
An eval is the closest thing an AI feature has to a test suite. It’s also the piece of the production-readiness stack teams skip most often, because it’s the least visible and the easiest to promise you’ll add later. Later usually arrives as a customer complaint.
Why you can’t unit-test a model
A unit test asserts that a function returns an exact value. Call it with 2 and 3, expect 5, fail on anything else. That works because the function is deterministic and “correct” is a single value.
A model breaks both assumptions. The same input can return different wording on two calls, so an exact-match assertion fails on output that was perfectly good. And “correct” is usually a judgment rather than a value. A support answer can be accurate, mostly accurate, or confidently wrong, and a summary can be faithful or subtly miss the point. There’s no single string to assert against.
So the test can’t be assert-equals. It has to be a graded check: run the model against inputs you care about, then score the output against criteria instead of against one right answer. That graded check is an eval, and running a set of them is an eval suite.
What an eval actually is
An eval has two parts: a dataset and a grader.
The dataset is a set of representative inputs, ideally drawn from real usage. The grader decides whether each output is good enough. Graders come in three shapes, and most production suites use all three depending on what’s being checked.
Programmatic grading is the cheapest and the most reliable. Does the output parse as valid JSON, is the returned category one of the allowed values, is the price a positive number, did the model stay under the length limit. Anything with a structurally correct answer gets checked in code, for free, with no ambiguity. This overlaps with output guardrails: the same validation that protects production at runtime also grades the model in testing.
Reference grading checks the output against a known-good answer. Does the response contain the facts it’s supposed to contain, does it avoid the ones it shouldn’t. You write the reference once, and every future run compares against it.
LLM-as-a-judge handles the open-ended cases no exact rule can score, like whether a rewritten paragraph kept the original meaning or whether a reply was actually helpful. A second model grades the first against a rubric. The technique is well established and genuinely useful, with one catch worth stating plainly: the judge needs its own calibration against human ratings, or you’ve moved the trust problem rather than solved it. A good practice is to have a different model do the grading than the one being graded, and to spot-check the judge against human scores periodically.
Where the eval set comes from
The best eval inputs are the ones that already broke. When a user sends something weird and the model handles it badly, that input becomes a permanent test case. The next prompt change has to survive it.
Beyond that, a useful set covers the spread of real inputs (the common cases, the long ones, the ones in another language, the ones that read as adversarial) plus the edge cases you hit during the build. The set grows over time the same way a regression suite grows: every bug you fix leaves a test behind so the bug can’t come back quietly. That accumulation is the point. A six-month-old eval suite is worth far more than the one you wrote on day one, because it encodes everything you’ve learned since.
What evals are actually for
Three jobs, and each one is a place teams get burned without them.
Catching regressions before users do. Change the prompt, the model, or a setting like temperature, run the suite, and see whether quality moved. Without evals, every change is a guess and every regression is invisible until someone outside the company notices. The gap between “we shipped a change” and “a customer told us it got worse” is where trust leaks out of an AI product. That covers the regressions you cause. The ones that arrive when you changed nothing, because a provider shipped a new model version or your traffic drifted, are the job of LLM observability: the same graded checks pointed at live production output on a schedule.
Making model swaps mechanical. This is the one most teams underuse. The model-agnostic pattern lets you swap the underlying model with a config change, but the pattern only pays off if you can tell whether the new model is good enough. The eval suite is what tells you. Point it at a cheaper model, and either it clears the bar or it doesn’t. The decision becomes a measurement instead of a debate, which is also how you take advantage of a new model the week it ships instead of the quarter you finally get around to testing it.
Calibrating guardrails. A guardrail tuned too tight rejects real users; tuned too loose it lets bad output through. You can only find the right setting by measuring both, so false-refusal rate belongs in the eval suite right next to accuracy. Tighten a filter, run the evals, and see whether you blocked more real attacks or just more real customers.
When evals are overkill
Evals are upfront work, and not every use of AI earns them. For a throwaway internal script where the model is incidental and a human reads every output anyway, a formal suite is more process than the task deserves. Build it crooked, look at the results yourself, move on.
The line is whether the feature is load-bearing and going to change. If real users depend on it, and the prompt or model will be edited more than once, “we test it manually” stops scaling at the second change. Manual testing checks the cases you remember to check, which are never the ones that break. And an eval suite isn’t free to keep either: it needs maintenance, and it’s only ever as good as the inputs you put in it. A suite that doesn’t include your hard cases will happily report that everything is fine.
The point
The reason evals feel optional is that a feature with no evals looks identical to a feature with great evals, right up until the day the model quietly gets worse. One team sees it in a number the same afternoon. The other hears about it from a customer weeks later and spends a day trying to figure out what changed.
That’s the whole value. An eval suite is the difference between changing your AI feature with confidence and changing it with your fingers crossed. It’s the first thing we build around a production AI feature that has to hold up, part of the standard scaffolding on every AI integration we ship, and the first thing we look for when we’re brought in to fix one that didn’t.
If you’re scoping an AI feature and trying to work out how you’ll know it’s still working six months in, tell us what you’re building. We respond within one business day with the gaps we see and a rough scope for closing them.