How to Build a Useful Test Set Without Fooling Yourself

The question

A test set is not a collection of examples. It is an argument about what you are willing to be wrong about.

Most test-set failures I have seen are not model failures. They are failures of the test set to represent the thing it claims to represent — and failures of the builder to notice that the set has quietly stopped being independent. The dangerous version is not a test set that produces a bad number. It is a test set that produces a good number for the wrong reason.

This post is about building the second kind as rarely as possible.

The short answer

Before you write a single example, decide which of three different claims your test set is meant to support. They require different sets, and conflating them is the root of most self-deception:

A set built for capability claims cannot support a distributional claim, no matter how many examples you add. Adding volume to a non-representative set produces a more precise wrong answer, not a better one.

The rest of this post is the checklist I use to keep those three claims separate — and to keep the set from becoming part of the training loop it is supposed to measure.

1. Compose deliberately, not by convenience

Microsoft’s evaluation-dataset guidance is the clearest published composition target I have found: roughly 60–70% common scenarios, 20–30% variations, 5–10% edge cases, and 5–10% adversarial cases, starting around 100 examples and expanding systematically.

The rationale for each bucket is the part that matters:

• Common scenarios establish the quality baseline. If these fail, nothing else is interesting.

• Variations prevent overfitting to specific phrasings. “Beginner trail” and “easy hike” should behave the same.

• Edge cases validate graceful degradation. Unusual inputs should not produce confident nonsense.

• Adversarial cases test safety and injection resistance. Deliberate misuse should not break the system.

Two details from that guidance are easy to skip and expensive to skip: edge cases and adversarial examples are usually better created manually than generated, because the model that generates them shares the blind spots you are trying to test; and production data rarely contains enough edge cases on its own, so sourcing from logs alone will systematically underweight the cases most likely to cause incidents.

A composition target is not a law. It is a commitment device. Write it down before you build the set, so that the final composition is a test result rather than an after-the-fact description.

2. Decide what “unseen” means before you build anything

Test-set leakage is not one failure. It is at least three, and they have different fixes.

Contamination is evaluation data present in pretraining. The NeurIPS analysis of this problem resolves an apparent paradox: leakage can be both potent and negligible depending on model capacity and the proportion of contaminated tokens. A model memorizes a benchmark based on its capacity and its incentive — the relative loss reduction from memorizing test data — so the effect scales with model size and with the dose of contamination, not with a simple “leaked / not leaked” switch.

Selection leakage is subtler and more common in engineering teams: you look at test-set performance, tune something, and the test set becomes an implicit part of training. The rule that follows is absolute in the MLOps literature — the evaluation set is held out permanently; updating it invalidates the metric. The moment you look at test-set performance and tune something, the test set is no longer a test set. Adding data to the test set “to improve representativeness” after seeing what the model got wrong is still leakage, because the new data was selected with knowledge of the failures.

Temporal leakage is the one that bites production systems. A test set constructed years after the data it resembles may not reflect the distribution the model will encounter, and the standard held-out assumption — that test and training come from the same distribution — is routinely violated in practice.

The practical rule: split once, before modeling, and treat the set as immutable. If a major distribution shift forces a new set, document the rationale, the date, and the data range, and treat the old and new metrics as not directly comparable.

3. Freeze it, hash it, and record the hash in the result

A frozen set that nobody can identify is not frozen in any useful sense.

The minimum discipline:

• Save the evaluation set to immutable storage at project start.

• Record a content hash of the set alongside every result.

• Use cross-validation on train/validation for all iteration; evaluate the frozen set once, at the end, on the final candidate.

This is the same record-keeping requirement from the reproducibility checklist, applied to the dataset rather than the prompt. A result without a dataset hash is a number with no provenance.

4. Match sample size to the smallest difference you care about

This is where most “model A beats model B” claims quietly collapse.

A single pass-rate with N runs has a 95% confidence interval half-width that depends on both N and the observed rate. For N=100 at a 90% pass rate, the half-width is roughly ±5.9 percentage points. At N=1,000, it narrows to about ±1.9 points. A two-point difference measured at N=100 is not a finding; it is noise with a decimal point.

The statistical literature is blunt about the small-N case. An ICML 2025 spotlight position paper argues that when LLM evaluation has fewer than a few hundred datapoints, confidence intervals based on the Central Limit Theorem severely underestimate uncertainty, and recommends Wilson score intervals or Bayesian credible intervals instead. The failure modes are concrete: when the observed rate is 0 or 1, the CLT interval collapses to a point; interval bounds escape the [0,1] range; and actual coverage falls far below the nominal 95%.

There is a paired-testing version of the same problem. An ICML 2026 analysis treats leaderboard gaps as paired hypothesis tests and shows that the common shortcut — taking a single-arm sample-size formula and multiplying by (1−ρ) as a “paired correction” — systematically underestimates the required sample size by half under small effects. On the Open LLM Leaderboard, 11 of 40 pairs were fundamentally unresolvable at conventional power; on MMLU-Pro top-10, 4 of 9 adjacent pairs were unresolvable, rising to 6 of 9 after accounting for multiple comparisons.

The practical translation: decide the minimum detectable effect before you run, not after you see the result. If you cannot afford the N that the effect size requires, the honest conclusion is “this test cannot resolve that difference,” not “the difference is real but small.”

5. Treat the scorer as part of the test set

A test set with an unstable scorer has an unstable result, and the instability does not appear in the score.

The empirical record on this is uncomfortable. A reproducibility note on LLM-as-judge graders found that a released evaluation harness invoked its grader without setting temperature or seed; the provider mapped the unset temperature to 1.0, so the grader ran unseeded at temperature 1.0, and items near the decision boundary flipped pass/fail across runs with per-item disagreement up to roughly 50% over 20 runs. Even at temperature=0, a Claude grader still flipped its verdict on 2 of 7 items.

Separately, work on serving nondeterminism found that with temperature zero, top-p one, and a fixed seed, accuracy across ten runs differed by 5 to 15 points on some tasks, and none of five hosted models gave repeatable accuracy on every task.

Two conclusions follow. First, a seed does not guarantee bitwise reproducibility on hosted endpoints; most commercial APIs are not reproducible even at temperature=0 with a fixed seed. Second, grader disagreement should be reported as a first-class health metric alongside the score, not buried as a caveat. If your scorer flips on 10% of items near the boundary, your pass rate has a floor of uncertainty that no amount of dataset expansion removes.

6. Report the composition and the failures, not just the score

A pass rate without composition is uninterpretable. A 92% on a set that is 90% common scenarios and 2% adversarial cases is a different claim than 92% on a set that is 40% edge cases — and the second is harder to achieve and more informative about production behavior.

The minimum reportable record for a test set:

json

{
  "dataset_id": "...",
  "dataset_sha256": "...",
  "n_examples": 400,
  "composition": {
    "common": 0.65,
    "variations": 0.22,
    "edge": 0.08,
    "adversarial": 0.05
  },
  "sources": ["production_logs", "manual_edge_cases"],
  "leakage_checks": ["pretraining_contamination_probe", "temporal_split", "no_test_set_tuning"],
  "scorer": "rule_based_v2",
  "scorer_nondeterminism_observed": true,
  "pass_rate": 0.912,
  "ci95": [0.884, 0.940],
  "ci_method": "wilson",
  "failures_captured": ["format_drift", "timeout", "empty_but_billed"],
  "not_tested": ["distributional shift beyond logged data", "adversarial families not in set"],
  "evidence_level": "reproducible experiment"
}

The not_tested field is the one that prevents the most overclaiming later. A test set defines a boundary. Naming the boundary is part of building the set, not an apology for it.

A worked example of why the boundary matters

The Gemini 4 Argon public benchmark record is a useful case study in test-set dependence, not because the model is good or bad, but because the same model produces very different numbers under different harness conditions.

On ARC-AGI-3, Argon reaches 97.5% in a provider-adapter harness that uses provider-specific features such as retained reasoning and context compaction, at a reported cost of $4,800. In the standard harness, which only allows simple notes between rounds, the same model reaches 49.8% at low reasoning, at a reported cost of $5,800. The harness is not a detail. It is part of the measurement, and the measurement changes by nearly a factor of two.

Separately, of the 19 benchmark rows in Google’s release table, 10 were calculated by Google itself and 9 came from public leaderboards, with different harnesses per row — and access to Argon was limited to a defender program at the time, so the numbers could not be independently reproduced outside Google’s partners and a small set of benchmark organizations.

The lesson for your own test set is not “benchmarks are unreliable.” It is narrower: the test conditions are part of the result, and a result without conditions is not a result. If your set does not record its harness, its composition, and its access condition, you have not measured the model. You have measured one path through it.

What this post does not prove

It does not prove that any particular composition ratio is correct for your task — 60/20/10/10 is a starting commitment, not a validated optimum. It does not prove that Wilson intervals are always the right choice; for paired comparisons, the paired design and its variance structure matter more than the interval formula. It does not prove that contamination is always material; the dose-response framing suggests it can be negligible in some regimes and potent in others, and which regime you are in is an empirical question about your model and your benchmark, not a general rule.

What it does establish is the structure of the self-deception problem: a test set fails quietly when it stops representing what it claims, when it stops being unseen, when its sample size cannot resolve the claim, or when its scorer is unstable. Each failure has a check. None of the checks is expensive relative to the cost of shipping on a number you cannot defend.

Reproduction notes

• Composition targets referenced: Microsoft evaluation-dataset guidance, 60–70% common / 20–30% variations / 5–10% edge / 5–10% adversarial, starting at ~100 examples.

• Freezing rule referenced: evaluation set held out permanently; split once before modeling; updating invalidates the metric.

• Small-sample interval guidance referenced: ICML 2025 position paper, Wilson or Bayesian intervals recommended below a few hundred datapoints.

• Paired-testing sample-size result referenced: ICML 2026, single-arm formula × (1−ρ) underestimates required N by half under small effects.

• Grader nondeterminism figures referenced: per-item disagreement up to ~50% over 20 runs unseeded at temperature 1.0; 2 of 7 items flipped at temperature=0.

• Serving nondeterminism figures referenced: 5–15 point accuracy spread across ten runs at temperature 0, top-p 1, fixed seed.

• ARC-AGI-3 harness comparison referenced: 97.5% provider-adapter vs. 49.8% standard harness at low reasoning, with reported costs.

For AI tool features, rules and availability, refer to their official sites.