API Contract Validation for Model APIs: What a 200 Does Not Prove

A regression suite of 500 API tests passes with green checks. Every test verifies status_code == 200. The production application begins crashing. Investigation reveals the provider silently changed a user_id field from an integer to a string. The existing tests missed this silent failure because they lacked deep contract validation — they were connectivity checks, not contract checks.

This is the foundational problem of API contract validation, and model APIs make it harder, not easier. A traditional API contract is written down: an OpenAPI document, a protobuf schema, a Pact file. A model API contract is partly written and partly assumed — the assumptions live in the integration code, the retry wrapper, the schema validator, and the operational budget. When those assumptions stop holding, the API often continues to return 200.

This post is the hub for a ten-part series on validating model API contracts. It defines what a contract is for a model-backed system, separates the four layers a 200 collapses, and links the instruments that make each layer checkable.

What “contract” means for a model API

In consumer-driven contract testing, a contract is the agreement between a provider and a consumer about the exact interface: request format, response schema, and status codes — verified without both sides deploying together. The provider commits to an interface; the consumer writes its expectations against that interface; and a contract test fails when the provider’s implementation drifts.

For a model API, the same definition applies, but the provider’s side of the agreement is a lifecycle, not a static document. The model ID may be pinned, but the parameter semantics, the per-request output ceiling, the caching behavior, the deprecation surface, and the billing structure can change between model generations. The consumer’s integration encodes assumptions about all of them. API contract validation is the practice of making those assumptions explicit, testing them, and recording the result so a reviewer can check whether the integration still means what it meant.

The scope is wider than schema validation. Schema validation is one gate. The contract includes the transport behavior, the protocol-level error taxonomy, the structural conformance of the output, the semantic admissibility of the values, and the operational properties — latency, cost, retry safety — that determine whether the integration can be depended on.

The four layers a 200 collapses

An HTTP 200 from a model API establishes four things: transport completed, the request was syntactically acceptable to the endpoint, a response object was produced, and the request was billable. It establishes nothing about completeness, schema conformance, semantic correctness, or operational reliability.

The contract validation problem is that application code routinely treats transport success as semantic success. The table below separates the layers and names the instrument that makes each one checkable.

A 200 clears L1 and L2. It says nothing about L3 or L4. Every contract validation instrument in this series exists because the layers below L2 are where the failures live, and they live there silently.

Why structured output is not a contract

The most common misconception in model API integration is that a response_format or response_schema is the contract.

It is not. Structured output shapes the generation; it does not guarantee the storage. The Gemini documentation is explicit that setting response_mime_type to application/json without a response_schema “acts only as a strong hint and carries a small risk of generating malformed JSON,” and that 100% valid JSON objects require both. Even with both, the documentation recommends a client-side JSON validator with a retry mechanism.

The gap has three sources. First, the schema is enforced generation-time, not insert-time: a conformant JSON object can still violate a database NOT NULL, a VARCHAR limit, or a foreign key. Second, a valid object can be operationally empty: a response with finish_reason: MAX_TOKENS can parse and carry no usable answer. Third, the generation schema and the storage schema are the same class only by coincidence: a field can be optional for generation and required for storage.

The empirical version of this gap is documented in tools built specifically for LLM output contracts. contractllm describes the production pattern it replaces — json.loads(response.choices[0].message.content) followed by a dict lookup and a float() cast — as three silent failure modes at the call site (JSONDecodeError, KeyError, ValueError). schemalock exists because “when you update a prompt or switch models, your downstream code can break silently”. prompt-contracts enumerates the failure classes: JSON breakage, missing fields, enum drift, performance regression, and model-switching behavior change.

Structured output is a request. The validation layer is the contract.

The instruments, and the evidence chain

The ten posts in this series are not a reading list. They are a verification sequence, and each link in the chain answers a question the previous one leaves open.

Start with the integration surface. The DeepSeek API Integration engineering guide covers authentication, timeout configuration, request logging, and error normalization — the contract layer at the boundary. It is the natural entry point because it establishes what the integration sends and what it receives before any semantic question is asked.

Then decompose the success response. What an API success response actually proves separates the four layers a 200 collapses. This is the conceptual pivot: once acceptance is distinguished from compliance, every subsequent instrument has a place to sit.

Then classify failures before retrying. Retries, timeouts, and the difference between a temporary failure and a bad request introduces the three-tier taxonomy — deterministic, transient, ambiguous — and the idempotency requirement that the ambiguous tier imposes. This is the operational contract’s first clause: a retry is only safe if the operation is idempotent, and the retry wrapper cannot know that on its own.

Then validate before the write. Add schema validation before model output reaches your database installs the structural gate: parse, schema, domain, and acceptance, with typed rejection reasons. This is the layer that prevents a refusal envelope from becoming a data record.

Then apply the same framework to specific models. The GLM 5.2 evaluation guide and the DeepSeek R1 evaluation guide are two applications of the same identity-verification and minimal-API-test pattern to different model families. GLM 5.2’s contribution is the identity contract — verifying the model identifier before trusting the output — and DeepSeek R1’s is the reasoning-model contract shift: thinking fields, concurrency limits, and the interaction between timeout budget and cost that a non-reasoning integration does not encounter.

Then test the version boundary. Version changes, deprecation notices, and compatibility assumptions introduces the three signal layers — silent, warning, hard — and the compatibility suite that would fail if an assumption stopped holding. The thinkingBudget case is the canonical example: a parameter that is a warning on one path and a hard 400 on another, with a retry wrapper that treats the 400 as transient and triples quota consumption.

Then measure the operational distribution. Measuring latency without hiding the slow cases and cost estimation when usage is uneven are the two operational contracts. Latency is a distribution with a censored tail; cost is a per-task quantity with thinking, retries, truncations, and cache storage as line items. A p50 without a timeout count and a cost per token without a cost per task are both claims that the record cannot support.

Then keep the comparison honest. How to report a model comparison without creating fake precision applies the resolution floor, the comparison interval, and the flip test. A correctly designed comparison can still be described in a way that makes the conclusion unsupported, and the reporting discipline is the last gate before the number becomes a decision.

Close with the release gate. The release gate turns the twelve instruments into twelve claims, each with a pass criterion, a waiver path, and an independent reviewer. The gate is the point where API contract validation becomes a shipping decision rather than a practice.

What API contract validation does not prove

It does not prove the output is correct. A response can pass transport, protocol, schema, domain, and acceptance gates and still contain a confidently wrong answer. Contract validation guards the shape and the operational properties, not the meaning.

It does not prove the model is reliable. A high validation-failure rate is evidence about the fit between the generation schema, the prompt, and the task — not a verdict on the model.

It does not replace load testing, evaluation suites, or human review. The contract is one layer of evidence; the accuracy, validity, completeness, and usability distinctions and the test-set leakage assessment answer the capability and validity questions that the contract instruments cannot.

What it does establish is narrower and necessary: the integration’s assumptions are explicit, tested, and recorded, so that a change in the provider’s behavior fails at the layer where the assumption broke rather than silently downstream. That is the difference between a 200 that means “the request was accepted” and a 200 that the application can act on.

A working demo is evidence. It is not a production system. An HTTP 200 is evidence of acceptance. It is not evidence of a contract — and API contract validation is the work of telling the two apart.

Frequently asked questions

What is API contract validation?

API contract validation is the practice of making an integration’s assumptions about a provider’s interface explicit, testing them, and recording the result. For model APIs, the assumptions include request parameters, error taxonomy, output schema, retry safety, latency, cost, and version behavior — not just the response schema. See What an API success response actually proves for the layer decomposition.

How is API contract validation different from schema validation?

Schema validation is one gate inside contract validation. It checks whether the output conforms to a declared structure. Contract validation also covers the transport layer, the protocol-level error classification, domain admissibility, retry idempotency, latency distribution, cost per task, and version compatibility. Add schema validation before model output reaches your database covers the structural gate in detail.

Does structured output guarantee a valid contract?

No. A response_format or response_schema shapes the generation; it does not guarantee storage conformance. A conformant JSON object can still violate a database constraint, and a response with finish_reason: MAX_TOKENS can parse while carrying no usable answer. Application-side validation is what makes non-conformant responses impossible to store.

What does an HTTP 200 from a model API actually prove?

It proves four things: transport completed, the request was syntactically acceptable, a response object was produced, and the request was billable. It does not prove the output is complete, schema-conformant, semantically correct, freshly generated, or the first attempt. The four-layer breakdown is in What an API success response actually proves.

Which model API errors are safe to retry?

Only transient failures — 500, 503, and rate-limit 429 — with backoff and jitter. Deterministic failures (400, 401, 403, 404, and quota-exhausted 429) will reproduce on retry. Ambiguous failures (timeouts, stream interruptions, empty 200s) are safe only if the operation is idempotent. The full taxonomy is in Retries, timeouts, and the difference between a temporary failure and a bad request.

How do I test whether a model API change breaks my integration?

Run a compatibility suite that would fail if an assumption stopped holding: sampling-parameter sensitivity, deprecated-parameter rejection, per-request output ceiling, cache behavior, and pinned model-ID resolution. Silent changes — parameters accepted but ignored — will not appear in a deploy log. The suite is documented in Version changes, deprecation notices, and compatibility assumptions.

Why does latency and cost count as part of the API contract?

Because they determine whether the integration can be depended on under production conditions. A p50 without the timeout count hides the censored tail; a cost per token without a cost per task undercounts thinking tokens, retries, and cache storage. Both are operational contract clauses. See Measuring latency without hiding the slow cases and Cost estimation when usage is uneven and inputs are messy.

Can I skip API contract validation for an internal tool?

You can reduce scope, but not skip the record. An internal tool can run on a smaller gate tier — for example, input boundary, decision records, and workflow reproducibility — while waiving latency, cost, and leakage rows with a recorded reason and a named owner. The tiering and waiver mechanism are in The release gate.

Start here

The series is ordered as a verification sequence, and the sequence starts at the integration surface:

→ DeepSeek API Integration: A Practical Engineering Guide

→ What an API Success Response Actually Proves

→ Retries, Timeouts, and the Difference Between a Temporary Failure and a Bad Request

→ Add Schema Validation Before Model Output Reaches Your Database

→ GLM 5.2: A Practical Evaluation Guide for Developers — and its reasoning-model counterpart, DeepSeek R1: A Practical Developer Evaluation Guide

→ Version Changes, Deprecation Notices, and Compatibility Assumptions

→ Measuring Latency Without Hiding the Slow Cases and Cost Estimation When Usage Is Uneven and Inputs Are Messy

→ How to Report a Model Comparison Without Creating Fake Precision

→ The Release Gate: Questions to Answer Before Shipping an AI Feature

For AI tool features, rules and availability, refer to their official sites.