Turning Production Traces Into a Regression Eval Set

What four vendor documentation pages specify about sampling and grading, and a five-step procedure for deciding what enters the suite and what gets retired

*Image: Software Testing Life Cycle by Perfect Happiness, licensed under CC BY-SA 4.0.*

Google's Gen AI evaluation overview lists three ways to build an eval dataset: upload a file of complete prompts, sample directly from production logs, or generate synthetic examples from a template. OpenAI's evaluation best practices guide puts a logging habit among its six tips and names one anti-pattern "biased design", described as eval datasets that do not faithfully reproduce production traffic patterns. LangSmith's dataset page documents an automation rule that adds a trace to a dataset when it matches conditions such as a low feedback score. Capture from production is documented on all three pages.

A set assembled only from inputs that broke once keeps passing after each fix lands. However many cases it holds, its pass rate then describes the history of the suite, and traffic the set has never seen goes unmeasured. That leaves one decision for whoever owns the suite: which traces earn a permanent place, which are held back, and which get retired.

Documentation was read on 2026-11-09. Eval tooling changes quickly, so re-check the linked pages before relying on a product name or a limit.

What the documentation pins down

Where cases come from. Google names production logs as a dataset source and frames the reason as evaluating real-world usage of the model. LangSmith builds a dataset from examples that hold inputs, outputs and optionally a reference output. Its page lists several ways to add them, including manual selection from a tracing project, an annotation queue, and an automation rule that fires when a trace meets a condition. OpenAI lists production data and historical log data among dataset sources, and flags the failure mode that makes this hard, a dataset that does not faithfully reproduce production traffic patterns.

What varies, by architecture. OpenAI's guide maps four architecture patterns to the place nondeterminism enters the system. A single model call varies in instruction following and functional correctness. A workflow repeats those two questions at each step. An agent adds tool selection and data precision, which ask whether the model chose the right tool and passed the right arguments. A multi-agent system adds agent handoff accuracy, meaning whether each agent recognises the point at which control should pass to another one.

What to grade with. Anthropic lists three grading methods with their tradeoffs and advises choosing the fastest, most reliable, most scalable one. Code-based grading is fastest and most reliable, and extremely scalable, though it lacks nuance for complex judgements. Human grading is the most flexible and highest quality, and also slow and expensive. LLM grading is fast, flexible and suited to complex judgement, and the page says to test its reliability before scaling it. Anthropic's design principles rank question count over grading quality: more questions with slightly lower-signal automated grading beat fewer questions with high-quality human grading. Its LLM grading tips ask for a detailed rubric, an empirical output such as correct or incorrect, and reasoning before the score.

What to hold back. Anthropic's workflow diagram tests a prompt against test cases during refinement, then against held-out evals before shipping. LangSmith supports named splits inside a dataset, so examples can be filtered by split as well as by dataset.

A five-step procedure

The steps below are my assembly of those parts. The sequencing, the sizing rules and the split scheme are mine, and each place where a documented claim gets extended is marked.

1. Capture the trace with an outcome attached. Store the input, the response, the tool calls with their arguments, and whatever judgement exists about whether the response was acceptable, along with who made that judgement. LangSmith's example shape is inputs, outputs and an optional reference output. Without an outcome the record says nothing about whether the behaviour should be repeated.

1. Report per failure mode and count each bucket. Track OpenAI's categories as separate counts instead of folding everything into one pass rate. A set where 280 of 300 cases are instruction-following failures is an invented illustration of the failure mode: it measures instruction following and tells you little about tool selection. Bucket counts also show which part of the system the suite has stopped watching.

1. Size each bucket for coverage. Anthropic's volume guidance is a single tradeoff between question count and grading quality. Applied per bucket it becomes a sizing rule, which is my extension. Twenty cases spread across five failure modes leave four per mode, in an invented illustration, and four cases will not measure a small change. Google, OpenAI and LangSmith each document synthetic data as one dataset source among several. Using it to fill a bucket that real traces leave short is my suggestion. Synthetic cases should carry a field that says so, as LangSmith does when it tags AI-generated examples as synthetic in their source metadata, because a synthetic case tests the behaviour whoever wrote it imagined.

1. Grade with code first and a judge second. Deterministic checks handle the cases whose answer is checkable: the tool name, the argument, the presence of a required field, the absence of a state change. Route the rest to a model grader with a detailed rubric and a pass or fail output. Before trusting that grader on new traffic, label a sample by hand and compare. OpenAI's guide warns about position bias from response order and verbosity bias toward longer answers, recommends pairwise comparison or pass or fail for reliability, and asks that response lengths be controlled.

1. Hold out, retire, and re-sample. In my scheme each case sits in one of three splits. The working split is what the team runs while editing the prompt and reviews each week. The held_out split stays out of prompt iteration, so the release number describes cases the prompt was not tuned on. The regression split holds cases the current system passes on every run: they still run in CI on every change, since that is what catches a return of the old behaviour, and they drop out of the weekly review. Cases also need a retirement path. When a deliberate product change makes the expected behaviour wrong, that case will fail for a reason nobody should fix, and keeping it in the run teaches the team to ignore a red result. Delete it and record why. OpenAI's continuous evaluation step asks for evals on every change and for the set to grow over time, which is the reason a model or prompt change should trigger a re-sample from new traffic.

A worked example

The record below is invented to show the shape. It is not a measurement from a real system.

{
  "id": "support-0137",
  "source_trace": "trace_8f21",
  "captured": "2026-11-09",
  "input": "can you cancel order 4417? tracking still shows it in the warehouse",
  "tools_available": ["lookup_order", "refund_order", "reset_password"],
  "observed": {"tool": "refund_order", "args": {"order_id": "4417"}},
  "outcome": "customer charged a restocking fee on an unshipped order",
  "verdict": "fail",
  "bucket": "tool_selection",
  "check": "tool_name == 'lookup_order' and state_change == false",
  "split": "held_out",
  "labelled_by": "human"
}

check is the part that runs without a person, so the case can sit in CI for a year. bucket keeps the per-failure-mode counts honest when someone adds twenty cases in a week. split takes one of the three values from step 5, and held_out keeps this case out of the loop where the prompt gets edited. Once the system passes it on every run, it moves to the regression split. The labelled_by field marks whose judgement produced verdict, which matters later, when a grader disagrees with the original label and someone has to decide which one changed.

Where the documentation stops

Criteria drift. A 2024 paper on LLM-assisted evaluation, Who Validates the Validators?, describes a problem the practice inherits from human review. People need criteria to grade outputs, and grading outputs helps them arrive at criteria, and some criteria appear dependent on the specific outputs observed. Reading outputs is therefore part of writing a rubric. What can be chosen is which outputs get read. My inference is that a rubric drafted from a batch of failures risks fitting that batch. Version the rubric alongside the dataset, and when revising it, read a fresh random sample of current traffic instead of the failures that prompted the revision, so new rubric items reflect what the system is producing now.

Traffic sampling and failure sampling pull in opposite directions. Sampling from production logs, one of the three dataset sources Google lists, reproduces real usage. My premise is that the cases most worth catching are rare in that usage, and no source read here gives a base rate for failing requests. Oversampling failures corrects the rare-case problem and, by my reading, reintroduces the mismatch OpenAI warns about. Keeping one traffic-weighted set and one failure-weighted set, reported separately, is a practical compromise. That compromise is my recommendation; the sources do not state it.

A generated rubric is model output too. The adaptive rubrics method, which Google recommends, generates a set of pass or fail rubrics per prompt and validates a response against each one, with an aggregated pass rate as the result. A generated test deserves the same validation as a generated answer, which means sampling the rubrics themselves and checking that each one tests what it claims.

Portability. OpenAI's guide states that Evals "will become read-only for existing users on October 31, 2026", that the platform is scheduled to shut down on November 30, 2026, and that existing content remains available during the transition window. A dataset that lives only inside a vendor console inherits that schedule. A versioned file in the repository does not, and it can be diffed when a case's expected behaviour changes.

The decision per trace

Three questions decide whether a trace joins the suite: does it carry a recorded outcome, does it name a failure mode, and can a script or a validated grader return pass or fail without a person. A trace that fails any of the three stays a log entry until someone supplies the missing part. In my judgement, the retirement and re-sampling steps matter more over a year than the size of the first capture, which only sets the starting count in each bucket. A suite that only holds old cases stops describing the traffic it faces.