How to Report a Model Comparison Without Creating Fake Precision

The question

A report says: Configuration B improves accuracy by 2.3 percentage points, from 85.0% to 87.3%.

What is wrong with that sentence?

Nothing about the arithmetic. The problem is that every number in it claims a precision the measurement does not have. The gap is 2.3 points — three significant figures on a difference that a 200-item test cannot resolve. The baseline is 85.0% — a decimal place that implies a hundredth-of-a-percent resolution the item count does not support. And the sentence presents a point estimate with no interval, which means the reader has no way to check whether the improvement exists at all.

Fake precision is not a rounding error. It is a claim of resolution that the measurement cannot support, and it is more dangerous than an obviously imprecise number because it is quotable. “B improves accuracy by 2.3 points” survives into a design document, a slide, and a release note, while the interval that would have shown the gap to be noise does not survive anywhere.

This post is about the six sources of fake precision in comparison reporting, the resolution floor that defines what precision is real, and the reporting template that keeps a correct comparison from becoming an incorrect sentence.

The short answer

Precision in a comparison report is real when three things are true: the number is representable at the measurement’s resolution, the uncertainty is reported alongside it, and the conclusion does not flip inside the uncertainty.

The three tests that catch all six:

1. Resolution test. Is the reported digit count representable given N? A 100-item binary test produces scores in whole percentage points. 87.3% is not a measurement on that test; it is a measurement on a 1,000-item test that was not run.

2. Interval test. Is the uncertainty reported? A gap without an interval is a preference with a percentage sign.

3. Flip test. Does the conclusion change at either bound of the interval? If yes, the conclusion is not supported at the reported precision, regardless of how many decimals the point estimate has.

The rest of this post is each source, the resolution floor, and the report that passes the three tests.

Source 1: spurious significant figures

The resolution floor of a benchmark is the smallest difference it can represent. For a binary-scored test with N items, the floor is 1/N — one item is one increment.

A 100-item test cannot produce 87.3%. The closest it can produce is 87% (87 items) or 88% (88 items). Reporting 87.3% is reporting a measurement from a test that was not run, and the extra digit is a statement about the reporter’s tooling, not the model’s performance.

The failure is more common than it looks, because the fractional number usually comes from one of three places, and only one of them is defensible:

• Partial-credit rubric. A scorer that awards 0.7 for a partially correct answer can produce 87.3% on a 100-item test. The number is representable, but its decimals now come from the rubric’s weights, not from the measurement. The weights are a choice, and the report has to say so.

• Averaging across runs. Averaging three runs of a 100-item test produces a number with more decimals and not more resolution. The item set is the same; the run-to-run variation adds noise, not resolution. A mean of three 87%/86%/89% runs is 87.33%, and the correct report is 87% ± the run-to-run spread.

• Unit conversion. Computing a percentage from a denominator other than the item count (weighted items, multi-part items) changes the resolution floor, and the new floor is the one that applies.

The rule: report the score at the resolution the test supports, and state the resolution. 87% (N=100; resolution 1 pp) is a measurement. 87.3% is a claim about a test that was not run.

Source 2: point estimates without intervals

A point estimate is a single draw from a distribution. The interval states the distribution’s plausible range, and without it the reader cannot tell whether the gap is a finding or a coincidence.

The interval arithmetic is the same as in the test-set post:

Below a few hundred datapoints, the normal approximation understates the uncertainty, per the ICML 2025 position paper referenced earlier; Wilson or Bayesian intervals are the correct choice, and the difference is largest near 0% and 100%, where the normal interval collapses to a point.

The reporting consequence is the comparison interval, not the individual intervals. If A is at 85% ± 4.2 and B at 87% ± 4.2, the gap is 2 points with an interval that depends on whether the comparison is paired. For a paired design the discordant-pair counts determine the interval, and the paired interval is usually narrower than the unpaired one — which is why the one-variable protocol records discordant pairs rather than aggregate rates.

The rule: report the interval on the comparison, not just on each score. A sentence with two intervals and no comparison interval leaves the reader to subtract one from the other, which is the arithmetic error that produces fake precision in the first place.

Source 3: composite indices

A composite index — a weighted average of sub-scores, sub-tasks, or sub-benchmarks — is where fake precision is manufactured most efficiently.

The mechanism is straightforward. Each component has a real interval. The composite has a point value and an interval that depends on the weights and on the correlation between components. Three things go wrong:

The weights are a choice, and the decimals inherit the choice. A composite of 0.5 × correctness + 0.3 × completeness + 0.2 × usability produces a number like 81.4. The 0.4 is a property of the weights (0.5, 0.3, 0.2 are themselves arbitrary to a decimal), not of the measurement. Change the weights to 0.4/0.35/0.25 and the composite moves, with no change to the underlying data. A number that moves when an editorial decision changes is not a precise measurement; it is a score with a parameter.

The composite interval is usually understated. If three components have ±5 pp intervals and are independent, the composite interval is about ±2.9 pp — narrower than any component. If they are correlated, which they usually are (a model that fails on correctness tends to fail on completeness), the composite interval approaches the component interval. Reports present the composite’s narrow interval without stating the correlation assumption, and the narrowness looks like precision. It is an assumption.

The composite hides the disagreement. Two configurations can have the same composite with opposite component profiles — A better on correctness, B better on usability. The composite says “equal.” The components say “different, and the difference matters for the decision.” Collapsing them removes the finding that the decision needs.

The rule: if a composite is reported, report the weights, the component intervals, and the correlation assumption. A composite without components is a number with a parameter. A composite with components is a summary of a measurement.

Source 4: ranks as measurements

A rank is a position in an ordered list. It is not a measurement of the model, and reporting it as one is fake precision in its purest form.

The ICML 2026 paired-testing analysis makes this quantitative. On the Open LLM Leaderboard, 11 of 40 pairs were fundamentally unresolvable at conventional power. On MMLU-Pro’s top-10, 4 of 9 adjacent pairs were unresolvable, rising to 6 of 9 after accounting for multiple comparisons. Those are large benchmarks with many items. A small benchmark makes the adjacent pairs less resolvable, not more.

A rank of #3 of 40 is a statement about the other 37 entries as much as about this one. If three entries are within noise, the rank is a coin flip that has been written down with a number. The rank also has no interval of its own, which means it presents the highest-precision-looking format (an ordinal integer) with the lowest-information content (a position inside an overlapping band).

The rule: report bands, not positions. Top 5, differences within noise is a usable claim. Rank #3 is usable only if the intervals for #2 and #4 exclude the entry’s interval — which is rarely the case and is checkable from the data the leaderboard already has.

Source 5: best-of-k selection

If k benchmarks are run and the best result is reported, the reported result is biased upward by roughly the maximum of k noisy draws. This is the winner’s curse, and it applies to internal evaluation as much as to published leaderboards.

The mechanism is arithmetic. With k = 5 benchmarks each with an interval of ±6 pp, the maximum is expected to exceed the true mean by a non-trivial margin purely by chance. The reported gain is real in the score and inflated relative to the true effect. The inflation does not appear in the report, because the report shows one benchmark.

The rule: report all benchmarks run, mark the primary as pre-registered, and treat the best of several as an estimate with an upward bias. A comparison that reports one of five is reporting a selection effect with a percentage attached.

Source 6: borrowed precision

A number can be precise and still be misleading, because its precision belongs to a context that did not travel with it.

The clearest case in the Argon benchmark record is ARC-AGI-3. The same model reaches 97.5% in a provider-adapter harness that uses provider-specific features such as retained reasoning and context compaction, at a reported cost of $4,800. In the standard harness, which only allows simple notes between rounds, the same model reaches 49.8% at low reasoning, at a reported cost of $5,800.

The 97.5% has three significant figures. It is a precise measurement of that harness. A reader who sees “97.5%” and carries it into a comparison against a number measured in the standard harness has imported the precision without the context, and the result is not a comparison — it is an arithmetic operation on two different measurements.

The related version is the aggregate row. Of the 19 benchmark rows in Google’s release table, 10 were calculated by Google itself and 9 came from public leaderboards, with different harnesses per row. A mean over those 19 rows is a composite (Source 3) whose components were measured under conditions the mean does not state.

The rule: a number’s precision is part of its measurement condition. Report the harness, and do not compare across harnesses without stating that the comparison is between conditions, not between models.

What precision is real

Precision is real when it is:

• Representable — at or above the resolution floor (1/N for binary, rubric granularity for weighted).

• Bounded — accompanied by an interval computed with a method appropriate to the sample size and the design.

• Decision-relevant — reported against a stated minimum detectable effect, so the reader knows what the test could have resolved.

• Flip-stable — the conclusion does not change at either bound of the interval.

That last test is the one that does the most work, and it is the one most reports skip. Apply it to the opening sentence: B improves accuracy by 2.3 points. Compute the conclusion at the lower and upper bounds of the comparison interval. If the lower bound includes zero, the conclusion at the lower bound is “no improvement,” and the sentence as written is not supported by the interval it omitted.

The flip test is cheap, and it converts a point estimate into a statement about what the data can adjudicate. It is the single most useful discipline in comparison reporting, because it is the test that makes the rest of the report checkable.

The reporting template

A comparison report that passes the three tests carries these fields:

json

{
  "comparison_id": "...",
  "test_date": "2026-10-20T14:00:00-05:00",
  "model_id": "gemini-4-argon",
  "harness_id": "standard_harness_v1",
  "n_items": 200,
  "resolution_floor_pp": 0.5,
  "primary_metric": "paired_pass_rate",
  "conditions": {
    "A": 0.850,
    "B": 0.873
  },
  "reported_precision": "nearest 0.5 pp",
  "gap_pp": 2.3,
  "gap_ci95_pp": [0.0, 4.6],
  "ci_method": "paired_wilson",
  "minimum_detectable_effect_pp": 3.0,
  "gap_exceeds_mde": false,
  "conclusion_flips_within_ci": true,
  "composite_reported": false,
  "rank_reported": null,
  "benchmarks_run": 1,
  "primary_preregistered": true,
  "harness_conditions_stated": true,
  "not_tested": ["concurrency effects", "alias drift"],
  "evidence_level": "reproducible experiment"
}

The four fields that carry the discipline:

• resolution_floor_pp states the smallest representable difference. reported_precision states that the score was rounded to it. Together they make Source 1 checkable.

• gap_ci95_pp is the interval on the comparison, not on each score. It makes Source 2 checkable.

• gap_exceeds_mde and conclusion_flips_within_ci are the flip test, stated as a result. In the example above, the gap is 2.3 points, the MDE is 3.0, and the interval includes zero — so the honest conclusion is “this test could not resolve a difference at the stated effect size.” That is a usable finding. “B is 2.3 points better” is not.

• harness_id and harness_conditions_stated make Source 6 checkable.

The template’s purpose is not completeness. It is to make the conclusion sentence derivable from the record. A reader who has the record should be able to write the sentence themselves and get the same one.

Failure cases observed

The three-significant-figure score. A 100-item evaluation reported 87.3% in a release note. The number was an average of three runs, which added decimals and not resolution. When a reviewer asked for the interval, it was ±8 points. The score was quoted in two subsequent design documents without the interval.

The composite with unstated weights. A model-quality index of 81.4, computed from a 0.5/0.3/0.2 weighting of three sub-scores. The sub-scores were 79, 86, and 82 — one configuration was better on the highest-weighted component and worse on the other two. The index reported a single number, the decision needed the profile, and the profile was not in the report.

The rank. A configuration at #3 of 40. The intervals for #2, #3, and #4 overlapped. The rank was quoted as evidence of a capability tier. The tier was the leaderboard’s composition, not the model’s performance.

The best of five. A team ran five benchmarks, reported the one with the largest gain, and attributed it to a prompt change. Re-running all five with a pre-registered primary showed the gain was within noise on four of them. The reported gain was the maximum of five noisy draws.

The borrowed 97.5%. A design document compared a local score against the ARC-AGI-3 provider-adapter figure, treating the harness difference as a rounding detail. The comparison was between two measurement conditions, and the report presented it as a comparison between two models.

The flip that was not checked. A comparison with a 2.3-point gap and a comparison interval of [0.0, 4.6]. The report stated the gap without the interval. At the lower bound, the gap is zero, and the decision the gap was cited to support was a coin flip. The flip test would have said so in one line.

None of these are model failures. All of them are reporting failures, and each maps to one of the six sources.

What this post does not prove

It does not prove that precise numbers are bad. A well-resolved score with a narrow interval, reported at its resolution, is exactly what a comparison should produce. The problem is precision that is claimed rather than measured.

It does not prove that every report needs every field. An internal screening comparison can report the score at the resolution floor and the gap with an interval, and omit the rest. The template is proportional to the decision the report supports, which is a judgment the post does not make for you.

It does not settle which interval method is correct for a given design. Wilson for single proportions, paired methods for paired designs, bootstrap for complex composites — the method has to match the design, and the record states which was used so a reader can disagree with the choice.

It does not resolve the interaction between precision and validity. A precisely reported comparison of a low-validity benchmark is a precise measurement of the wrong construct, and the accuracy-vs-validity post is the relevant frame. Precision is necessary for a usable comparison. It is not sufficient.

What it does establish is the structure of the problem: fake precision is a resolution claim the measurement cannot support, and it is produced by significant figures, missing intervals, composites, ranks, selection, and borrowed context. The three tests — resolution, interval, flip — catch all six, and the flip test is the one that converts a point estimate into a statement the data can defend.

A working demo is evidence. It is not a production system. A precise-looking number is evidence of formatting. It is not evidence of precision — and the flip test is how you tell the difference.

For AI tool features, rules and availability, refer to their official sites.