Retries, Timeouts, and the Difference Between a Temporary Failure and a Bad Request

The question

A retry wrapper is the smallest piece of code in a model-backed system, and the one most likely to turn a temporary failure into a permanent one.

In my first-contact test on Argon, a client with automatic retries reported a 99% success rate. The raw logs showed a 6% first-attempt failure rate absorbed by retries at a cost the summary never mentioned. Both numbers were true. Only one was useful — and the one that was reported described the retry layer’s success, not the endpoint’s reliability.

This post is about the decision the retry layer is making, usually implicitly: is this failure a temporary condition that a retry can resolve, or a bad request that a retry will reproduce? Getting that distinction wrong is not a performance problem. It is a correctness problem, a cost problem, and — when the retry lands after a partial write — a data problem.

The short answer

Not all failures are retryable, and the HTTP status code alone does not tell you which. Three tiers, and only the middle one is a retry:

The third tier is where production incidents live, and it is the tier the standard retry wrapper — “retry on 5xx and timeout, fail on 4xx” — handles worst, because the ambiguous cases either look like 2xx (empty output) or like 1xx/5xx (timeout, interruption) without enough context to decide.

The rest of this post is the taxonomy, the ambiguous cases, and the record that makes the retry decision auditable after the fact.

What the error envelope actually carries

A Gemini API error response carries a status code, a canonical error status string, and a message. The status code is for control flow. The message is for humans. Neither is sufficient on its own, and the gap between them is where retry bugs come from.

The 400 row deserves a specific note for Argon. Supplying deprecated sampling parameters in later model generations returns HTTP 400. Today those parameters are silently ignored, so the same request returns 200. A retry wrapper that treats all 4xx as deterministic is correct here — but only if it treats 400 as deterministic now, before the transition. A wrapper that treats 400 as “maybe transient” will burn quota on a request that will never succeed.

Tier 1: the failures a retry cannot fix

These are the cases where the request itself is wrong. Retrying is not a recovery strategy; it is a way of paying for the same error twice.

Observed on Argon: a request with max_output_tokens above the per-request streaming ceiling (65,536) returns HTTP 400. The request is structurally invalid for the call path. A retry produces the same 400 at the same cost.

Observed on Argon: a request containing a deprecated parameter returns 200 today and 400 in a later generation. The first case is worse, because it does not fail — it succeeds while doing something other than what the code asked. This is the success-response problem in a specific form: a 200 that proves acceptance, not compliance.

Observed on Argon: malformed response_format schemas return 400. The schema is checked at request time. A retry with the same schema reproduces the error.

The rule for Tier 1 is not “don’t retry 4xx” as a blanket policy. It is “classify the 4xx before deciding.” A 400 caused by a deprecated parameter and a 400 caused by a transient serialization bug in a client library look identical at the status-code level, and they require opposite decisions. The message is the only discriminator, and messages are not stable across versions. This is why the error classification should be recorded as a field, not inferred at decision time.

Tier 2: the failures a retry can fix, with backoff

These are the cases where the request is valid and the environment is not.

500 INTERNAL. The server failed while processing a valid request. Retry with exponential backoff and jitter.

503 UNAVAILABLE. The service is temporarily unable to handle the request. Retry with backoff. Note that 503 also appears in the ambiguous tier when it interrupts a streaming response — see below.

429 RESOURCE_EXHAUSTED. This status conflates two conditions that require opposite responses:

• Rate limit (per-minute): the request rate exceeded the window. Retry with backoff, honoring the retry-after value if present.

• Quota exhaustion (per-day): the daily quota is gone. Retry does nothing until reset. Backing off and retrying a quota-exhausted 429 is the classic thundering-herd pattern — the retries increase load on a system that is already at capacity, and they consume the client’s own retry budget for no benefit.

The status code does not distinguish them. The response body or the account dashboard does. A retry wrapper that treats every 429 as “back off and try again” is correct for one condition and harmful for the other.

The backoff parameters matter more than the retry policy. Exponential backoff with full jitter is the standard, and the reason is empirical: without jitter, retries from multiple clients synchronize into waves that reproduce the original rate limit. The specific constants should be set from the observed rate-limit window, not copied from a default.

Tier 3: the ambiguous cases

This is the tier the standard retry wrapper gets wrong, and the tier that produces duplicate rows.

Case 1: timeout after the request was sent. The client gives up waiting. The server may have processed the request, may be processing it, or may have discarded it. A retry is safe only if the operation is idempotent — and a generation that will be written to a database is not idempotent by default.

Case 2: stream interruption. The client received some tokens, then the connection dropped or the server returned an error mid-stream. The server may have billed for the tokens it generated. The retry produces a second generation, and the application may now have a partial first output and a complete second output, with no way to tell which one to keep.

Case 3: 200 with an empty output. This is the case the retry wrapper never sees, because it is a 200. The response is empty or truncated and still billed. The retry layer does not fire. The success-checker accepts it. The validation layer, if one exists, is the only line of defense — and if validation is absent, an empty record is stored with a 200 in its provenance.

Case 4: 503 interrupting a long-thinking request. At thinking_level=high, a request can run for minutes. A 503 mid-request may arrive after the model has generated and billed substantial thinking tokens. The retry produces a second generation and a second bill. The first attempt’s cost is paid whether or not the output is used.

The common structure in all four cases: the failure signal does not carry enough information to decide whether the retry is safe. The decision requires context the HTTP layer does not have — whether the operation is idempotent, whether the first attempt was billed, whether a partial result was already persisted.

The idempotency problem

The retry decision assumes the retry is safe. It is safe only if the operation is idempotent.

For a read — generate and discard, generate and display without storing — the operation is idempotent in effect. A duplicate generation costs money and latency, but it does not corrupt state.

For a write — generate and store, generate and send, generate and act — the operation is not idempotent. A retry after a successful write produces a duplicate. A retry after a partial write produces a truncated record plus a complete one. A retry after a timeout whose first attempt actually succeeded produces two complete records with different content, and the application has no way to know which one to keep.

In my testing, the API does not provide a client-supplied idempotency token that the server uses to deduplicate requests. Deduplication must be application-side. That means:

• A logical request ID assigned before the first attempt, carried through every retry, and stored with the result.

• A uniqueness constraint on that ID at the storage layer, so the second write is rejected rather than duplicated.

• A check-before-write for the ambiguous cases, where the application queries whether the logical request already produced a stored result before retrying.

This is not a model property. It is application architecture. The retry wrapper cannot supply it, because the wrapper does not know whether the operation is a read or a write.

For AI tool features, rules and availability, refer to their official sites.