Writing Evals for AI Features: A Testing Guide for Before You Ship

A checkout flow either charges the right amount or it doesn’t. An AI feature that summarizes a support ticket can produce a dozen different, equally correct summaries for the same input. That difference is why teams that test AI features with ordinary unit tests end up either drowning in false failures or, more often, just not testing the AI part at all. Evals are the actual answer, and building a basic one takes an afternoon, not a new department.

Why a unit test doesn’t work here

A unit test asserts one specific output for a given input. That works when there’s exactly one correct answer: a function that adds two numbers, an endpoint that returns a specific status code. An LLM call rarely has exactly one correct answer. Ask it to summarize the same support ticket ten times and you’ll get ten different, plausible summaries, all of them arguably correct, none of them byte-for-byte identical.

Write a unit test that checks for exact string equality against one of those ten outputs, and it fails on the other nine, not because the feature is broken, but because the test is asking the wrong question. The right question isn’t “did it produce this exact text.” It’s “did it satisfy the requirements a good summary has to satisfy,” which is a rubric, not an equality check.

The three parts every eval needs

A test set. A collection of real or realistic inputs the feature will actually encounter. Twenty to fifty examples pulled from real usage, support tickets, or known edge cases is enough to start. Resist the temptation to invent synthetic examples from scratch; a small set built from cases that have actually happened catches more real regressions than a large set of hypothetical ones.

A scoring method. How you decide whether each output passed. This can be rule-based (does the output contain a required field, stay under a length limit, avoid a banned phrase), model-graded (a second LLM call scores the output against a rubric), or human-graded (someone reads the output and rates it). Most practical eval suites mix all three: cheap rule-based checks for anything mechanical, model grading for anything that needs judgment, and human review for a small sample to keep the model grader honest.

A pass threshold, decided in advance. Before you run the eval, decide what pass rate is acceptable. Deciding this after seeing the results is how a team talks itself into shipping a regression because the number “still looks pretty good.”

A minimal eval harness, in code

Here’s a working structure using rule-based checks plus a model-graded rubric, the pattern that covers most real AI features without needing a dedicated framework yet.

# eval_harness.py
import json
from dataclasses import dataclass

@dataclass
class EvalCase:
    input: str
    must_include: list[str]      # rule-based check
    rubric: str                  # what a grader model should verify

def load_cases(path: str) -> list[EvalCase]:
    with open(path) as f:
        raw = json.load(f)
    return [EvalCase(**case) for case in raw]

def rule_based_check(output: str, case: EvalCase) -> bool:
    return all(phrase.lower() in output.lower() for phrase in case.must_include)

def model_graded_check(output: str, case: EvalCase, grader_call) -> bool:
    prompt = f"""Rubric: {case.rubric}
Output to grade: {output}
Does the output satisfy the rubric? Answer only "yes" or "no"."""
    verdict = grader_call(prompt).strip().lower()
    return verdict.startswith("yes")

def run_eval(cases: list[EvalCase], feature_call, grader_call) -> dict:
    results = []
    for case in cases:
        output = feature_call(case.input)
        passed_rules = rule_based_check(output, case)
        passed_rubric = model_graded_check(output, case, grader_call)
        results.append({
            "input": case.input,
            "passed": passed_rules and passed_rubric,
            "output": output,
        })
    pass_rate = sum(r["passed"] for r in results) / len(results)
    return {"pass_rate": pass_rate, "results": results}

A test set file looks like this:

[
  {
    "input": "Customer says their order arrived damaged and wants a refund.",
    "must_include": ["refund", "damaged"],
    "rubric": "The summary must identify the customer's request (refund) and the reason (damaged item), in two sentences or fewer, without inventing details not present in the original message."
  }
]

Running this against every prompt change or model swap, and failing a CI step if pass_rate drops below your threshold, turns “did that prompt edit break anything” from a question someone has to remember to check manually into something the build answers automatically.

The part that actually needs care: trusting the grader

Model-graded evals scale further than a human reading every output, but the grader is itself an LLM call and can be wrong. It can be too lenient, marking borderline outputs as passing to avoid a negative-sounding judgment. It can be inconsistent across nearly identical inputs. It can share the same blind spots as the model it’s grading, if both are similar enough to make the same category of mistake.

The fix isn’t to avoid model grading, it’s to validate it before trusting it. Take a sample of 15 to 20 cases the grader scored, review them by hand, and check whether your judgment matches the grader’s. If it disagrees on more than a small fraction, tighten the rubric (vague rubrics produce vague grading) or add more specific pass/fail criteria before wiring the grader into a CI gate that can block a deploy. This validation step is easy to skip under deadline pressure, and skipping it is exactly how a team ends up with an eval suite that reports a comfortable 95% pass rate while shipping a feature that’s actually failing in ways nobody’s checking for.

Evals belong in CI, not a pre-launch checklist

The highest-value moment for an eval suite isn’t the week before a big launch, it’s every single prompt tweak, model version bump, or system message edit afterward. These changes are exactly the ones that silently regress a specific case without breaking anything obvious in casual testing: a prompt edit that improves tone but drops a required disclaimer, a model upgrade that handles the common case better but starts hallucinating on a rare one.

Wiring the eval script from above into a CI step, the same way a project already gates on its regular test suite, catches that class of regression at the same point in the workflow: before merge, not after a user reports it. If your team is building or hardening an AI feature and testing hasn’t kept pace with how fast the prompts and models are changing, that gap is usually the first thing worth closing, not the last.

Where to start this week

Pick one AI feature already in production. Pull twenty real inputs from actual usage or support tickets. Write a rubric for what a correct output looks like, get specific enough that two people reading it would grade the same output the same way. Wire up a script like the one above, run it, and look at what fails. It will not be a clean pass rate the first time, and that’s the actual value: a fast, repeatable way to see exactly where the feature breaks before a user finds it for you.