The Release Gate: Questions to Answer Before Shipping an AI Feature

The question

What has to be true before an AI feature ships?

The common answer is a checklist of concerns: have we tested it, do we have monitoring, is there a rollback, did legal sign off. Every item on that list is reasonable, and a release that satisfies all of them can still fail on the first user who sends a malformed input. The checklist asks whether the team considered the concerns. It does not ask whether the team can produce the evidence that the concerns are handled.

That is the difference this post is about. A release gate is not a list of things to think about. It is a set of claims, each with a stated piece of evidence, each with a pass criterion that can be checked by someone who was not in the room.

The short answer

The most common release-gate failure is not a failed test. It is a claim that was never made explicit, and therefore never had an evidence requirement.

A team that says “the feature is ready” has made an implicit claim. The gate’s job is to make the claim explicit, one clause at a time, and to require the evidence that would let a reviewer disagree. The format is the same for every clause:

Claim — what the team asserts is true.
Evidence — what would show it.
Pass criterion — what a reviewer checks.
Owner — who is answerable when it is absent.

A gate with this structure can be audited. A gate without it produces a release that “passed” and a postmortem that cannot say what was checked.

The four claims, and why they cannot be collapsed

Every AI feature release makes four claims. They are the same four from the prototype boundary post, and they remain distinct at the gate because each fails differently and each requires different evidence.

The gate’s first job is to refuse the substitution. A demo is evidence of feasibility and not of the other three. A benchmark score is evidence of correctness and not of reliability. A dashboard is evidence of observability and not of operability. Each column of the table is a different artifact, and a release that supplies one artifact is making one claim.

The gate

The table below is the gate I use. Each row is a claim, the evidence that answers it, the pass criterion a reviewer applies, and the post in this series that documents the instrument. The rows are ordered by the cost of discovering the absence late.

Twelve rows. A reviewer who can check eleven and not the twelfth has found the gate’s weak point, and that is the gate working — the purpose is to locate the missing evidence, not to collect signatures.

The gate is proportional, and the tier decides which rows are mandatory

A twelve-row gate applied uniformly is theater in the other direction: a low-stakes internal tool forced through a customer-facing process will have its evidence fabricated to clear the queue. The gate has to be proportional to the cost of being wrong, and the proportionality is a tier decision made before the gate is run, not a negotiation during it.

The waiver is not a loophole. A waived row requires a recorded reason and a named owner, so that the absence of the evidence is a decision rather than an omission. The difference between a T2 release that waived row 10 and a T2 release that forgot row 10 is the record, and the record is what a postmortem reads.

The tier itself should be assigned before the feature is built, because the tier determines what the build has to produce. A team that discovers at release time that it needed the compatibility suite did not build the compatibility suite, and the discovery converts into a deferred risk rather than a gate.

The release record

The gate’s output is not a sign-off. It is a record that a reviewer can check, and that a postmortem can read.

json

{
  "release_id": "extraction-v2.3.0",
  "feature": "invoice field extraction",
  "tier": "T2",
  "gate_run_at": "2026-10-20T14:00:00-05:00",
  "model": {
    "model_id": "gemini-4-argon",
    "pinned": true,
    "alias_used": false,
    "call_surface": "interactions.create",
    "thinking_level": "medium",
    "max_output_tokens": 8192,
    "sdk": "google-genai 2.30.1"
  },
  "claims": {
    "input_boundary": {
      "evidence": "validation_gate_v2; rejection_reasons enum",
      "pass": true,
      "owner": "extraction-team"
    },
    "acceptance_vs_compliance": {
      "evidence": "per_request_record with finish_reason and params_effective",
      "pass": true,
      "owner": "extraction-team"
    },
    "retry_classification": {
      "evidence": "error_tier at decision time; idempotency key on writes",
      "pass": true,
      "owner": "platform"
    },
    "decision_records": {
      "evidence": "three gate queries answerable",
      "pass": true,
      "owner": "platform"
    },
    "latency_distribution": {
      "evidence": "p99 client-observed at concurrency 20; censored timeouts reported",
      "pass": true,
      "owner": "extraction-team"
    },
    "cost_per_task": {
      "evidence": "measured at introductory and standard rates; amplifiers counted",
      "pass": true,
      "owner": "finance-eng"
    },
    "version_compatibility": {
      "evidence": "compatibility_suite_v1.2; 6 assumptions checked",
      "pass": true,
      "owner": "platform"
    },
    "quality_claim": {
      "evidence": "N=400 frozen set; gap CI excludes zero; flip test stable",
      "pass": true,
      "owner": "validation"
    },
    "score_interpretation": {
      "evidence": "construct stated as operationalized task; composition reported",
      "pass": true,
      "owner": "validation"
    },
    "leakage_assessment": {
      "evidence": "contamination check performed; selection procedure pre-registered; temporal channel not measured",
      "pass": "partial",
      "waiver_reason": "temporal channel not applicable to task type",
      "owner": "validation"
    },
    "human_review": {
      "evidence": "sampling design documented; gold-set agreement 0.86; rejections routed",
      "pass": true,
      "owner": "review-ops"
    },
    "workflow_reproducibility": {
      "evidence": "clean-checkout run; run record retained",
      "pass": true,
      "owner": "extraction-team"
    }
  },
  "waived_rows": ["leakage_assessment.temporal"],
  "not_tested": [
    "concurrency above 20",
    "cross-region routing",
    "adversarial families outside the frozen set"
  ],
  "decision": "ship_with_stated_limits",
  "reviewer": "independent_of_build_team",
  "evidence_level": "release_gate"
}

Four things make this record usable:

claims is keyed by claim, not by document. A gate that is keyed by artifact — “the test report,” “the monitoring doc” — cannot say which claim the artifact supports. The claim key forces the mapping.

pass can be partial with a waiver_reason. A gate that permits only binary outcomes will be gamed into binary outcomes. A recorded partial pass is a decision; a silent pass is an omission.

not_tested is a required field. The test-set post made the same argument for benchmark records: the boundary is part of the measurement. A release record without a boundary is a claim of universality.

reviewer is independent of the build team. This is the only field that makes the rest checkable. A gate run by the team that built the feature measures the team’s confidence; a gate run by someone with no stake measures the evidence.

The failure cases that justify each row

The gate’s rows are not hypothetical. Each one corresponds to a failure documented in this series, and the failure is the argument for the row.

Rows 1–2: the empty 200. A long-decode request exceeded the streaming per-request ceiling and returned a response that was empty or truncated while still being billed. The status was 200. A client that checked the status accepted the output, and an empty record was stored with a 200 in its provenance. The failure was discovered when a downstream report showed a gap with no corresponding error log. Row 1 would have rejected the record; row 2 would have made the acceptance decision visible.

Row 3: the invisible retry. A wrapper with automatic retries reported a 99% success rate over a 6% first-attempt failure rate. The 200s were real. The success rate described the retry layer, not the endpoint, and the cost and latency of the absorbed failures were not in any report. Row 3 makes the retry decision — and its cost — a recorded event.

Row 5: the healthy p50. A dashboard showed p50 = 1.8s while users complained about multi-minute waits. The p50 was computed over all configurations; the high-thinking requests were 8% of traffic and the entire source of the complaints. A separate report showed p99 = 4.2s with a 3% timeout rate excluded from the percentile calculation by construction. Row 5 requires the distribution, the censored population, and the segmentation.

Row 6: the single-rate estimate. An annual cost estimate computed entirely at the introductory $2/$10 rates while seven weeks of the year fall after the December 1 transition to $4/$20. The understatement was roughly 45%, and it was entirely in the price table. A second failure in the same row: a cost model that counted visible answer tokens and missed thinking, which on a classification task undercounted the output line by more than an order of magnitude.

Row 7: the carried-forward parameter. A migration kept thinkingBudget, which Argon does not support. The migration guide described it as deprecated with a warning; the observed behavior included hard 400 errors, and the retry wrapper treated the 400 as transient, tripling quota consumption and delaying the fix by a release cycle. A separate case: max_output_tokens carried forward unchanged produced finish_reason: MAX_TOKENS with an empty response on complex prompts — the thinking budget consumed the entire allowance before the answer began.

Rows 8–9: the 2.3-point improvement. A release note reported configuration B as improving accuracy by 2.3 points, from 85.0% to 87.3%. The test had 200 items. The comparison interval was [0.0, 4.6]. The minimum detectable effect was 3.0 points. The flip test said the lower bound includes zero, so the honest conclusion was this test could not resolve a difference at the stated effect size — and the sentence that shipped said the opposite.

Row 10: the overlap-clean benchmark. An n-gram check found no exact overlap. The benchmark items had been paraphrased in a public dataset that was in the training corpus. The contamination was real, the detector was defeated by the paraphrase, and the accuracy drop on a rephrased evaluation was 6.8 points — measured after the decision.

Row 11: the rubber-stamp queue. A 100% review requirement, two reviewers, 300 daily items. Review duration converged to under 20 seconds per item and the rejection rate to 0.3%. A gold-set audit found the reviewers missed 40% of injected errors. The step was staffed, logged, and produced a signature. It did not produce a gate.

Row 12: the kernel-state workflow. A notebook “converted to a script” by copying cells in order failed on its first clean run, because a variable was defined in a cell that appeared later in the document but had been executed earlier. The fix added the missing definition to the top, which worked and hid the fact that the notebook’s output had depended on an execution order nobody had written down.

Twelve rows, twelve failures. None of them is exotic. Every one of them shipped, or nearly shipped, on a release that had passed a checklist of concerns.

For AI tool features, rules and availability, refer to their official sites.