The Reproducibility Checklist for Model Testing

The question

If I hand you a test result for a model-backed system, what would you need in order to check whether I am right?

That is the whole reproducibility question, and it is easier to answer than most teams expect. It is also the question most demo-grade testing quietly skips. A result that cannot be re-run is not evidence about the model. It is evidence about one afternoon.

This post is the checklist I use before I will treat a model test as citable — by me, by a colleague, or by a reader who wants to disagree with me in writing.

The short answer

Reproducibility for model testing has three levels, and they are not the same claim:

Most published “model tests” stop somewhere inside R1 and call it R3. That gap is where confident wrong answers come from.

One necessary caveat before the checklist: models are not deterministic, and treating them as deterministic is itself a reproducibility error. Reproducibility does not mean “get the same string.” It means “get a comparable distribution, and show the distribution.” Any checklist item that assumes one run equals the result is the wrong item.

The checklist

1. Freeze the identity of the model

• □

  Record the exact model ID as sent on the wire: gemini-4-argon.

• □

  Record whether that ID is a stable version or an alias that can move underneath you.

• □

  Record the access tier (free / paid / preview / allowlisted) and any regional routing.

• □

  Record the SDK and its version: google-genai>=2.30.1 (Python), @google/genai>=2.29.1 (JavaScript).

• □

  Record the call surface: interactions.create, or models.generate_content, streaming or not.

Why: the same model ID through a different call surface can have different per-request output ceilings. In my first-contact test, non-streaming direct HTTP capped at 32,768 tokens per request and streaming at 65,536, with continuation required beyond that. That is a property of the path, not the model, and it silently shapes your results.

2. Freeze the time

• □

  Test date, with time zone.

• □

  Duration of the run.

• □

  Any known provider-side change window overlapping the run.

Why: model behavior and availability are time-indexed facts. “Argon did X” without a date is not a finding; it is a rumor with a citation format.

3. Freeze the parameters that still exist — and strip the ones that don’t

• □

  Record thinking_level (low / medium / high) — on Argon this is the real control surface, default medium.

• □

  Record max_output_tokens and confirm it covers thinking plus visible output.

• □

  Record whether temperature, top_p, top_k were sent, and record that they were ignored.

• □

  Remove deprecated sampling parameters from the harness rather than relying on silent ignoring.

Why: a harness that sets temperature=0 and reports “deterministic results” is reporting a fiction. On Argon those parameters are ignored today and may return HTTP 400 in a later generation. If your reproducibility story depends on them, you do not have a reproducibility story.

4. Freeze the prompt as rendered, not as authored

• □

  Hash and store the fully rendered prompt: system instruction, user turn, tool definitions, retrieved context, few-shot examples.

• □

  Store the retrieval step’s inputs and outputs if RAG is in the path.

• □

  Record token counts for input, thinking, and output separately.

Why: almost every “I can’t reproduce it” case I have seen is a prompt-assembly difference, not a model difference. The template was the same; the retrieved chunk was not.

5. Freeze the dataset

• □

  Version the input set with a content hash.

• □

  Record size, sampling method, and inclusion criteria.

• □

  Check for test-set leakage against anything in the prompt, the retrieval index, or the fine-tuning data.

• □

  Store a fixed subset for smoke re-runs and keep the full set for the published result.

Why: a dataset that changes between runs makes every downstream number uninterpretable. Version the inputs before you trust the outputs.

6. Choose a sample size on purpose

• □

  State the decision the test is meant to inform.

• □

  Choose N from the smallest difference you need to detect.

• □

  Report the uncertainty, not just the point estimate.

Approximate 95% confidence interval half-width for a single pass rate:

Why: with N=100, a 90% result and a 95% result are not distinguishable. Many published “model A beats model B” claims live entirely inside that overlap.

7. Store raw responses, including the failures

Raw response bodies, not just extracted fields.

Failure samples in full — the malformed JSON, the truncated stream, the empty-but-billed response.

Error codes, retry counts, and final status per request.

Usage and cost fields per request, including cache-related fields.

Why: summaries hide the failure modes that matter most. The empty response billed as complete is the single most useful artifact from my first-contact test, and it would have vanished from a pass-rate table.

8. Record the cost and the clock

Total and per-request cost, with the pricing snapshot and date.

Latency distribution, not the mean alone: p50, p90, p99, and the slowest cases.

Why: a benchmark without cost and latency is a capability claim pretending to be an engineering result. Argon’s introductory pricing through 2026-11-30 ($2/$10 per 1M input/output) differs from the post-introductory $4/$20, so a cost figure without a date is not reusable.

9. Publish the scoring code

The exact scorer, including normalization and partial-credit rules.

Whether scoring was model-assisted, and if so, which model and which prompt.

A human spot-check sample with agreement rate.

Why: scorer changes move results more than model changes do. An unstated scorer is an unstated variable.

10. State what you did not test

Explicit out-of-scope list.

Known confounds.

Conditions under which the conclusion would not hold.

Why: this is the item teams skip and reviewers value most. It is also the cheapest defense against overclaiming later.

Minimum viable record

If the full checklist is too much for a given decision, do not skip reproducibility — reduce scope and keep the record complete for that smaller scope. The minimum I will accept as citable:

json

{
  "model_id": "gemini-4-argon",
  "call_surface": "interactions.create",
  "sdk": "google-genai 2.30.1",
  "access_tier": "paid",
  "test_date": "2026-10-20T14:00:00-05:00",
  "thinking_level": "medium",
  "max_output_tokens": 8192,
  "sampling_params_sent": ["temperature", "top_p", "top_k"],
  "sampling_params_effective": false,
  "prompt_sha256": "...",
  "dataset_sha256": "...",
  "n_runs": 400,
  "pass_rate": 0.912,
  "ci95": [0.884, 0.940],
  "failures_captured": true,
  "cost_usd_total": 0.00,
  "latency_p50_ms": 0,
  "latency_p99_ms": 0,
  "not_tested": ["throughput under load", "structured-output schema conformance at scale"],
  "evidence_level": "reproducible experiment"
}

Twelve fields. That is the floor. If a result cannot produce this block, it is a note, not a test.

Failure cases I have actually seen

The determinism illusion. A harness pinned temperature=0 and reported identical outputs. On Argon those parameters are ignored, so the “determinism” was never real — the runs happened to agree because the task was easy. The test would have reported stability right up until it reported an incident.

The moving alias. A comparison ran across two weeks against a model ID that was not version-pinned. Some of the measured difference was the model, and some was the calendar. The result was unfalsifiable, which felt like rigor and was actually the absence of it.

The invisible retry. A script with automatic retries reported a 99% success rate. The raw logs showed a 6% first-attempt failure rate absorbed by retries at a cost the summary never mentioned. Both numbers were true; only one was useful.

The unversioned dataset. A “regression suite” gained three examples mid-project. Every before-and-after number in the report was computed on a different set. Nobody had changed the model.

None of these are model failures. All of them are record failures.

Where reproducibility is not worth the cost

A checklist applied without judgment becomes theater. Match the record to the decision:

• Personal prototype, no users: keep dates, model ID, and raw failures. Skip the statistics.

• Internal tool, small team: add prompt hash, dataset hash, N, and the failure log.

• Anything reaching production or a published claim: full record, CI, published limitations.

The point is not to make every experiment expensive. It is to make the cost of the record proportional to the cost of being wrong.

What this checklist does not prove

It does not prove your test measures what you intended — construct validity is a separate problem, and a perfectly reproducible test of the wrong metric is still wrong. It does not prove the result generalizes beyond the dataset, the access tier, or the date. It does not protect against a scorer that is consistently incorrect.

It does one thing: it makes disagreement possible. A reader who cannot re-run your test cannot check your claim, and a claim that cannot be checked is not yet an engineering result.

Reproduction notes

• Model: gemini-4-argon

• Access: paid Gemini API tier

• Test date: 2026-10-20

• Python SDK: google-genai>=2.30.1; JavaScript SDK: @google/genai>=2.29.1

• Confidence intervals computed as Wald intervals for a single proportion; for N<100 or pass rates near 0 or 1, use Wilson intervals instead

• Pricing snapshot used for cost fields: introductory $2/1M input, $10/1M output through 2026-11-30; $4/$20 thereafter

The next post in this series applies this record to a concrete case: what an API success response actually proves, and what the missing fields above look like when they are absent from a production incident.

For AI tool features, rules and availability, refer to their official sites.