Measuring Latency Without Hiding the Slow Cases

The question

A latency dashboard shows p50 = 1.8s and p99 = 4.2s. The service looks healthy. Users are complaining.

What did the dashboard measure?

Most latency reports answer a different question than the one the incident is asking. They measure the latency of the requests that completed, at the concurrency the test was run, on the call surface that was instrumented, with the retries counted as part of the response or excluded from it — and the report rarely states which of those choices was made. Each choice hides a different set of slow cases, and the slow cases are the ones that produce incidents.

This post is about the ten mechanisms that hide slow cases in a latency report, the latency structure specific to a reasoning model, and the reporting discipline that keeps a p50 from becoming a claim about behavior it never observed.

The short answer

Latency is a distribution, not a number. The mean is the least informative summary of a distribution whose shape is the thing that matters, and a p50 alone is a statement about the median user that says nothing about the users who generate the incidents.

Three rules, before the mechanisms:

1. Report the distribution, not the summary. p50, p90, p95, p99, p99.9, and the count of requests above the timeout budget.

2. Report the censored cases. A request that timed out does not have a completion latency. Excluding it from the calculation lowers the reported distribution by exactly the amount that matters.

3. Report the conditions. Concurrency level, call surface, thinking level, streaming mode, cache state, and whether retries are included.

The rest of this post is why each of those rules exists, and what the report looks like when they are followed.

The ten hiding mechanisms

Every one of these is a routine practice. None of them is dishonest. Together they produce a latency report that describes a system no user experiences.

1. Reporting the mean

The mean is pulled by the tail and says nothing about either the median or the tail. A mean of 2.1s can describe a distribution where 90% of requests complete in 1.5s and 1% take 60s. The mean is arithmetically correct and operationally useless.

What it hides: the shape. A distribution with a thin tail and one with a fat tail can have the same mean and completely different incident profiles.

2. Reporting p50 only

p50 is the median user. It is the user who does not file the incident. A service with p50 = 1.5s and p99 = 45s has a median that looks fine and a tail that produces the support tickets.

What it hides: the users who experience the slow cases, which is the population the incident is about.

3. Excluding timeouts from the distribution

A request that timed out at 30s has no completion latency. If the report computes latency over completed requests, the 30s cases are removed from the sample — and they are removed precisely because they were slow. The reported p99 is computed over a population that excludes the slowest requests by construction.

This is right-censoring, and it biases the reported distribution downward. The correct treatment is to report the timeout rate as a separate metric alongside the latency distribution, and to report a lower bound on the timed-out latency (the timeout value itself) so that the tail is not invisible.

What it hides: the requests that were so slow they did not produce a latency measurement.

4. Measuring at low concurrency

A single-request test measures the latency of the system when no one else is using it. Production runs at N concurrent requests. Queueing, rate-limit backoff, and provider-side capacity effects all change the distribution, and they change it most in the tail.

The difference is not a constant offset. At low concurrency, p99 may be near p50. At high concurrency, the tail expands because requests queue behind each other, retries multiply under rate limiting, and the slowest dependency becomes the bottleneck for the whole chain.

What it hides: the concurrency-dependent expansion of the tail.

5. Aggregating across configurations

A p50 computed over thinking_level=low and thinking_level=high requests is a weighted average of two different distributions. The low-thinking requests are fast and numerous; the high-thinking requests are slow and few. The aggregate p50 describes the low-thinking population, and the high-thinking tail is diluted by the majority.

What it hides: the per-configuration distribution. Latency must be reported by configuration, because the configuration is the thing that determines the distribution.

6. Measuring only first attempts

A retry succeeds on the second attempt at 2.0s. The first attempt timed out at 30s. The client-observed latency is 32.0s. A report that measures only successful completions records 2.0s and loses the 30s that the user actually waited.

This is the invisible retry in latency form. The success rate hides the retry; the latency report hides the wait.

What it hides: the time the user spent waiting for the retry to complete.

7. Not separating cached from uncached

Implicit caching applies to shared prompt prefixes of 6,144 tokens or more, and a cache hit is materially faster than a fresh generation. A latency distribution that mixes cached and uncached requests is a mixture of two distributions with different shapes. The cached requests pull the median down; the uncached requests produce the tail.

What it hides: the latency of the requests that had to do the work.

8. Reporting server-side timing only

The provider’s reported latency and the client-observed latency differ by network time, SDK overhead, and retry wait. For a single-hop request the difference is small; for a multi-step agentic workflow it is not.

What it hides: the latency the user experiences, which is the one that determines whether the workflow is usable.

9. Reporting TTFT as total latency for streaming

Time-to-first-token (TTFT) is the latency until the first token arrives. Total generation time is the latency until the response completes. For streaming responses, TTFT can be short while total generation is long, and reporting TTFT as “latency” describes the responsiveness of the stream, not the duration of the task.

What it hides: the total wait, which for a reasoning model with long output is the number that matters.

10. Coarse histogram buckets

A histogram with buckets [0–1s, 1–5s, 5–30s, 30s+] cannot resolve a p99. The 30s+ bucket is where the incident lives, and a count in that bucket does not tell you whether the tail is at 31s or 300s.

What it hides: the resolution of the tail. If the histogram is the only record, the tail is not measurable after the fact.

The latency structure of a reasoning model

Argon changes the latency shape in ways that make the hiding mechanisms more consequential, not less.

Thinking is latency before output. The model reasons before it answers, and thinking tokens are billed as output. The visible answer is preceded by a generation phase that produces no user-visible tokens. For a non-streaming request, the user sees nothing until the answer is complete, and the thinking time is inside the wait.

Thinking level dominates the distribution. thinking_level ∈ {low, medium, high}, default medium. High-effort reasoning can run for minutes on a complex prompt. A latency report that does not segment by thinking level is reporting a mixture in which the slow population is defined by the configuration.

The per-request ceiling interacts with latency. Non-streaming direct HTTP caps at 32,768 tokens per request; streaming at 65,536; beyond that, long decode continues across requests. A request near the ceiling has a longer expected duration than one well below it, and the timeout budget has to be set against the configuration, not the model ID.

Thinking level and output ceiling share a budget. The comparison analysis showed that thinking_level and max_output_tokens interact: high thinking under a modest output ceiling produces truncation and, in latency terms, a request that spends time thinking and then fails to produce a usable result. The latency of that request is real and its outcome is a failure, which is exactly the category a mean-only report loses.

Streaming changes what “latency” means. For streaming requests, TTFT and total generation time are separate metrics with separate user experiences. A workflow that needs the answer to complete has the total as its latency. A workflow that can act on partial output has the TTFT. Reporting one as the other is the ninth hiding mechanism.

What to measure

The minimum latency record for a model-backed system:

The paired comparison is the reporting rule: report the distribution alongside the timeout rate and the retry-inflated fraction, segmented by configuration. A p99 without those companion numbers is a claim about a population that excludes the requests that did not complete and the wait that was spent retrying.

What to report

A practical latency record:

json

{
  "window": "2026-10-20T00:00:00-05:00/2026-10-20T23:59:59-05:00",
  "model_id": "gemini-4-argon",
  "call_surface": "interactions.create",
  "streaming": true,
  "thinking_level": "medium",
  "max_output_tokens": 8192,
  "concurrency": 20,
  "n_requests": 10000,
  "n_completed": 9871,
  "n_timeout": 129,
  "timeout_budget_ms": 60000,
  "ttft_ms": {"p50": 0, "p90": 0, "p99": 0},
  "total_latency_completed_ms": {
    "p50": 0, "p90": 0, "p95": 0, "p99": 0, "p99_9": 0, "max": 0
  },
  "total_latency_client_observed_ms": {
    "p50": 0, "p90": 0, "p95": 0, "p99": 0, "p99_9": 0, "max": 0
  },
  "retry_inflated_fraction": 0.0,
  "cache_state": "split",
  "cache_hit_share": 0.0,
  "censoring_note": "timeouts excluded from completed percentiles; lower bound on censored latency = timeout_budget_ms",
  "not_measured": ["concurrency above 20", "cross-region routing"],
  "evidence_level": "runtime"
}

Four fields carry the integrity:

• n_timeout and timeout_budget_ms together state the censored population and the lower bound on its latency. A report without both has silently removed the slowest requests.

• total_latency_client_observed_ms states the wait including retries. Reporting only the completed latency is the sixth hiding mechanism in numerical form.

• retry_inflated_fraction states what share of the user waits included a retry. A p99 client-observed that is much higher than the p99 completed is the retry layer showing up in the latency.

• concurrency states the load at which the distribution was measured. A distribution at concurrency 1 is not evidence about concurrency 20, and the record should not pretend otherwise.

The not_measured field is the same discipline as the not_tested list in the test-set post. A latency report defines a measurement condition. Naming the conditions that were not measured is part of the report, not an apology for it.

The failure cases observed

The healthy p50. A dashboard showed p50 = 1.8s. Users complained about multi-minute waits. The p50 was computed over all configurations, and the high thinking requests were 8% of traffic and the entire source of the complaints. The aggregate p50 described the low-thinking majority; the tail described the population the incident was about.

The censored tail. A latency report showed p99 = 4.2s and no mention of timeouts. The system had a 3% timeout rate at a 30-second budget. The 3% were excluded from the percentile calculation because they had no completion latency. The true p99, including the timed-out requests at their lower bound, was above the budget. The report described the 97% of requests that completed and called the result “latency.”

The retry-inflated wait. A client-observed latency of 32 seconds for a request whose completed latency was 2 seconds. The first attempt timed out at 30 seconds; the retry succeeded. The success-rate report showed a success. The completed-latency report showed 2 seconds. Neither showed the 32 seconds the user waited. The record that would have shown it is total_latency_client_observed_ms plus retry_inflated_fraction, and neither was being collected.

The low-concurrency test. A load test at concurrency 1 produced p99 = 3.1s. Production at concurrency 50 produced p99 = 22s. The tail expansion was not linear in concurrency; it was driven by rate-limit backoff and queueing. The test was not wrong about the system at concurrency 1. It was wrong as evidence about the system at concurrency 50.

The cache mixture. A latency report over all requests showed p50 = 1.1s. The cached requests were 70% of traffic and completed in under a second. The uncached requests had p50 = 3.8s and a much fatter tail. The aggregate p50 described a population that did not exist for any individual request — every request was either cached or uncached, and the aggregate described neither.

The TTFT reported as latency. A streaming integration reported “latency” as the time to first token. It was 400ms. The total generation time for the same requests was 12 seconds. The report was accurate about responsiveness and misleading about duration, and the workflow that depended on the complete answer was being planned against the wrong number.

None of these are model failures. All of them are measurement failures, and the measurement is where the hidden slow cases live.

What this post does not prove

It does not prove that every system needs every metric. A synchronous internal tool with a small user population can run on p50 and a timeout count. The metrics should be proportional to the cost of a slow case in your system, which is a judgment the post does not make for you.

It does not prove that a particular percentile is the right one. p99 is conventional; p99.9 is appropriate when the tail carries the incidents; p95 is defensible for a low-stakes tool. The post argues for reporting the distribution and the censored population, not for a specific cut.

It does not address cross-region or cross-provider latency, which introduce a network dimension the single-environment test did not exercise.

It does not address the latency of agentic multi-step workflows, where the total latency is a sum over steps and the per-step distributions interact. That is a separate post.

What it does establish is the structure of the hiding problem: a latency report hides the slow cases when it excludes them from the population, aggregates them into a majority, measures them under conditions that do not occur in production, or reports a summary that cannot express the shape. The record above is the correction, and the correction is what makes the p50 checkable rather than merely quotable.

A working demo is evidence. It is not a production system. A fast p50 is evidence about the median request. It is not evidence about the users who file the incidents.

Reproduction notes

• Model: gemini-4-argon

• Access: paid Gemini API tier

• Test date: 2026-10-20

• SDK: google-genai>=2.30.1 (Python)

• Call surfaces exercised: interactions.create (streaming and non-streaming); models.generate_content non-streaming for the ceiling and timeout cases

• Thinking behavior: thinking_level ∈ {low, medium, high}, default medium; high-effort runs observed in the minutes range on complex prompts; thinking tokens billed as output

• Ceilings observed: 32,768 tokens per request non-streaming direct HTTP; 65,536 streaming; continuation required beyond that

• Caching: implicit only, prefix threshold 6,144 tokens, persistent by default; explicit context caching not supported

• Concurrency in the production-shaped run: 20 concurrent requests; low-concurrency comparison at 1

• Timeout budgets exercised: 30s and 60s against the same configuration; the 30s budget produced the timeout-then-retry case

• Censoring treatment: timed-out requests excluded from completed percentiles and reported separately with the timeout budget as a lower bound

• Pricing snapshot: introductory $2/1M input, $10/1M output through 2026-11-30; $4/$20 thereafter

• Failure samples retained: aggregate-p50 configuration mixture; censored-tail report; 32s client-observed / 2s completed retry case; 1-vs-50 concurrency gap; cache-mixture p50; TTFT-as-latency report

The next post in this series takes the cost side of the same operational question: cost estimation when usage is uneven and inputs are messy — because a cost model that assumes uniform requests has the same shape of error as a latency report that assumes uniform requests.

For AI tool features, rules and availability, refer to their official sites.