Comparing Model Configurations One Variable at a Time

The question

When you compare two model configurations, what has to be true before the difference you measured is the difference you think you measured?

“Change one variable at a time” is the standard answer, and it is a good starting point. It is not sufficient on a non-deterministic system, and it fails in a specific and predictable way: it assumes the variables are independent, and model configurations are not. The most consequential comparison errors I have reviewed were not uncontrolled experiments. They were controlled experiments on the wrong variable, or controlled experiments where the control was fictional.

This post is the comparison protocol I use, including the part where single-variable comparison stops being enough.

The short answer

Single-variable comparison is a screening method, not a conclusion method. It has three jobs and one limitation:

The limitation: when two factors interact, a one-at-a-time comparison can rank them in the wrong order. The correct protocol is OFAT to screen, then a targeted joint test on the two factors most likely to interact — because that interaction test is usually the one that changes the decision.

The rest of this post is what “controlled” actually requires before OFAT is even meaningful.

What counts as a variable

A model comparison has more variables than the configuration object suggests. The ones that silently differ between two runs are the ones that produce false conclusions.

Any two of these that differ between your “A” and “B” conditions are part of the comparison, whether you intended them to be or not. The freeze list exists to make that set explicit.

The fake variables

Before the protocol, a warning about variables that appear controlled and are not.

On Argon, temperature, top_p, and top_k are ignored, and supplying deprecated parameters in later model generations returns HTTP 400. If your comparison harness varies temperature between conditions, you have not varied a variable. You have created a control that does not control anything, and the result will show “no difference” — which is arithmetically correct and scientifically meaningless, because the factor had no effect by construction.

This is worse than a missing control, because it looks like a finding. “We tested temperature 0.2 vs 0.8 and saw no difference” is the kind of sentence that gets quoted in a design document. On Argon, that sentence describes the API’s behavior, not the model’s.

The related trap is the deprecated-parameter transition. A parameter that is silently ignored today can return HTTP 400 in a later generation. A comparison recorded in October against a parameter that becomes an error in December is not reproducible, even if every other field was frozen. Record params_sent and params_effective as separate fields, per the per-request log in the success-response analysis.

The freeze list

Before the first run, write down what will not change. This is the same discipline as the reproducibility checklist, applied to the comparison rather than the single result.

• Model ID and version (or alias policy, stated explicitly)

• Call surface, streaming mode, and SDK version

• Access tier, region, and run window

• Prompt template, hash of the rendered prompt, and every few-shot example

• Response schema and its version

• Retrieval index and its version, if any

• Dataset and its content hash

• Scorer and its version

• Sampling parameters sent, and which were effective

• max_output_tokens, and whether it is being varied or held

• Concurrency level and timeout configuration

If a field cannot be frozen — date, access tier, provider-side behavior — record it as a known non-frozen variable and say so in the result. A comparison with one documented non-frozen variable is usable. A comparison with an undocumented one is not.

Define the decision metric before the run

A comparison needs one primary metric that the decision depends on, and any number of secondary diagnostics.

• Primary metric: the one that decides A vs. B. It must be defined before the run, including the threshold for “better.”

• Secondary diagnostics: cost, latency, failure modes, validation-failure rate. These inform the decision but do not replace it.

• Minimum detectable effect: the smallest difference you care about. This determines N, per the test-set composition guidance.

The order matters. If you choose the metric after seeing the results, you will choose the one that favors the configuration you already preferred. That is not a hypothetical failure; it is the default behavior of a human looking at a results table.

A concrete example of a decision-ready primary metric:

For the same 200-item frozen set, configuration B is preferred over configuration A if its paired pass rate is higher by at least 3 percentage points, with the paired 95% confidence interval excluding zero, at equal or lower p95 latency.

That sentence is testable, falsifiable, and decided in advance. Most comparison documents I see cannot produce it.

Pair the samples

Aggregate pass rates hide the pairing. The same 200 items should be run through both configurations, and the outcome should be recorded per item, not just per configuration.

The reason is statistical and practical. Paired designs are more sensitive when the two configurations agree on most items — which is the common case for configurations that differ in one parameter. The unpaired shortcut is also where the sample-size estimate goes wrong: the common practice of taking a single-arm formula and applying a “paired correction” underestimates the required N by roughly half under small effects, per the ICML 2026 analysis referenced in the test-set post. If you are comparing configurations that differ by a few points, the paired design and its variance structure matter more than the interval formula.

Practical rules:

• Record per-item outcome: pass, fail, and the failure type.

• Record the discordant pairs: items where A passed and B failed, and the reverse. Those counts, not the aggregate rates, are the primary evidence.

• Randomize or counterbalance run order if provider-side time effects are plausible.

• Run the comparison in one session where possible; if not, record the split.

Where OFAT breaks: the worked example

Here is the case that changed my protocol.

I compared thinking_level at low and high while holding everything else at the defaults — including max_output_tokens. On the surface, a clean single-variable comparison.

The result was that low looked better on the primary metric. A one-variable reading says: use low.

That reading is wrong, and the reason is structural. On Argon, max_output_tokens covers thinking plus the visible answer, and thinking is billed as output. At a modest output ceiling, high reasoning consumes more of the budget before the answer is complete, so a higher proportion of its responses are truncated. The comparison was not measuring thinking level. It was measuring thinking level under an output ceiling that penalizes thinking.

The fix is a small joint test. A 2×2 matrix:

The single-variable comparison of thinking_level produced the ranking A > B. The joint test produced the ranking B2 > A2 ≈ A1 > B1. The two-factor conclusion is different from the one-factor conclusion, and the one-factor conclusion was not wrong about what it measured — it was wrong about what the measurement meant.

This is not an argument against OFAT. It is an argument about what OFAT is for. It screens. It does not conclude when the factors plausibly interact.

The general rule: the two factors most likely to interact in a model configuration are the ones that share a resource. thinking_level and max_output_tokens share the output budget. Prompt length and max_output_tokens share the context window. Concurrency and timeout share the latency budget. When two factors draw on the same resource, assume interaction until the joint test says otherwise.

The interaction screening protocol

A practical sequence that keeps the matrix small:

1. Freeze the baseline. Pick the configuration currently in use, record the freeze list, and run the full dataset once as the reference.

2. OFAT screen. Vary one factor at a time against the baseline. Keep the screen small — three to five factors, coarse levels, the primary metric only.

3. Identify shared resources. From the screen, list the factors that plausibly draw on the same budget, ceiling, or rate.

4. Run the joint test on the top pair. A 2×2 is usually enough. Add a third factor only if the 2×2 shows an interaction and you need to locate it.

5. Decide with the paired result, not the screen result. The decision record cites the joint test, and the screen is reported as screening.

6. Report the interaction, not just the winner. “B is better” is an incomplete conclusion if B is only better above a ceiling that the application cannot always afford.

Step 6 is the one that gets dropped in practice, and it is the one that prevents the next incident. A configuration chosen at a 16,384 ceiling that later runs at a 2,048 ceiling because of a cost constraint is a regression nobody will attribute to the original comparison.

Failure cases observed

The confounded upgrade. A team changed model version and prompt template in the same release, then measured an improvement. The improvement was real; the attribution was not. When the prompt change was reverted on the new model, half the gain disappeared. Two variables, one measurement, no attribution.

The fake variable. A harness compared temperature=0 against temperature=0.7, found no difference, and concluded the model was insensitive to sampling. On Argon both settings were ignored. The conclusion described the API, not the model.

The ceiling interaction. The worked example above, encountered twice in different teams, with the same initial wrong conclusion.

The harness change. A comparison ran low on the interactions.create surface and high on models.generate_content non-streaming. The per-request output ceilings differ (65,536 streaming vs. 32,768 non-streaming). The measured difference mixed reasoning level with call-path capacity, and the document reported it as a reasoning-level result.

None of these are model failures. All of them are comparison-design failures.

The comparison record

The minimum record for a citable comparison:

json

{
  "comparison_id": "...",
  "test_date": "2026-10-20T14:00:00-05:00",
  "model_id": "gemini-4-argon",
  "sdk": "google-genai 2.30.1",
  "access_tier": "paid",
  "call_surface": "interactions.create",
  "dataset_sha256": "...",
  "n_items": 200,
  "primary_metric": "paired_pass_rate",
  "minimum_detectable_effect_pp": 3.0,
  "conditions": {
    "A": {"thinking_level": "low", "max_output_tokens": 2048},
    "B": {"thinking_level": "high", "max_output_tokens": 2048},
    "C": {"thinking_level": "low", "max_output_tokens": 16384},
    "D": {"thinking_level": "high", "max_output_tokens": 16384}
  },
  "params_sent": ["thinking_level", "max_output_tokens"],
  "params_effective": ["thinking_level", "max_output_tokens"],
  "discordant_pairs": {"A_pass_B_fail": 0, "A_fail_B_pass": 0},
  "paired_ci95": [0.0, 0.0],
  "interaction_observed": true,
  "decision_basis": "joint_test",
  "cost_usd_total": 0.0,
  "not_tested": ["concurrency effects", "alias drift over time"],
  "evidence_level": "reproducible experiment"
}

Two fields carry the comparison’s integrity. discordant_pairs is the primary evidence for a paired design; the aggregate rates are derived from it, not the other way around. decision_basis states whether the conclusion rests on the screen or the joint test, which is the difference between a comparison and a screening result presented as a comparison.

What this protocol does not prove

It does not prove the winning configuration is the best available. OFAT plus one interaction test explores a corner of the space, not the space. A configuration that loses in the 2×2 may win in a region the matrix never sampled.

It does not prove the result generalizes across model versions, access tiers, or dates. Those are non-frozen variables, and the record should say so.

It does not prove the primary metric is the right one. A perfectly paired comparison of the wrong metric is a precise wrong answer. The decision metric is a judgment, and the protocol only makes it an explicit, pre-registered judgment rather than an after-the-fact one.

It does not resolve the small-sample problem for genuinely small effects. If the difference you care about is under two points and N is 200, the honest conclusion is that this test cannot resolve it — not that the configurations are equal.

What it does establish is the discipline that makes a comparison citable: one variable screened at a time, the shared-resource pairs tested jointly, the pairing recorded per item, and the decision basis stated explicitly. That is the difference between a measurement and a preference with a table attached.

A working demo is evidence. It is not a production system. A clean single-variable comparison is evidence of attribution. It is not evidence that the variable was the one that mattered.

Reproduction notes

• Model: gemini-4-argon

• Access: paid Gemini API tier

• Test date: 2026-10-20

• SDK: google-genai>=2.30.1 (Python)

• Call surfaces exercised: interactions.create (streaming and non-streaming); models.generate_content non-streaming for the harness-change case

• Factors varied in the worked example: thinking_level ∈ {low, high}; max_output_tokens ∈ {2,048, 16,384}

• Factors confirmed ignored: temperature, top_p, top_k

• Observed per-request ceilings: 32,768 tokens non-streaming direct HTTP; 65,536 streaming; continuation required beyond that

• Pricing snapshot: introductory $2/1M input, $10/1M output through 2026-11-30; $4/$20 thereafter

• Paired sample-size reference: ICML 2026 paired-testing analysis, single-arm formula × (1−ρ) underestimates required N by roughly half under small effects

• Failure samples retained: truncated responses at the 2,048 ceiling under high; 200-with-empty-output at stream ceiling; 400 on out-of-range max_output_tokens

For AI tool features, rules and availability, refer to their official sites.