A Prototype Is Not a Production System: The Missing Questions

A prototype answers one question: can this work at all?

A production system answers a different set of questions: will this work for every user, every time, under load, when a dependency fails, when the input is wrong, when the person who built it is on vacation, and when someone audits it eighteen months from now?

The gap between those two question sets is not a matter of polish. It is a matter of categories. Prototypes and production systems fail in different ways, for different reasons, against different standards. Treating a working prototype as "90% done" is the most reliable way to ship something that looks finished and behaves unpredictably.

This post is about the questions that get skipped — the ones that don't appear in a demo, don't show up in a happy-path test, and don't become visible until the system meets a distribution it was never designed for.

What a prototype actually proves

A prototype proves feasibility. It shows that under controlled conditions — one user, one input shape, one environment, one dependency state — the system can produce an acceptable result.

That is a real and necessary milestone. It is also a narrow one. It says nothing about:

• Consistency under input variation. A prototype sees the inputs its author chose. Production sees the inputs real users produce, including malformed, empty, adversarial, and unexpected ones.

• Behavior under load. A prototype handles one request at a time. Production handles concurrent requests, queuing, rate limits, and contention.

• Failure recovery. A prototype has no recovery path because its author is watching. Production needs retries, fallbacks, circuit breakers, and a way to fail without corrupting state.

• Operability. A prototype has no logs, no metrics, no alerts, no runbook. When it breaks, someone has to reconstruct what happened from memory.

• Ownership. A prototype belongs to its builder. Production belongs to a team, a process, and an on-call rotation.

None of these are optional in production. All of them are invisible in a demo.

The missing questions

The questions below are the ones I have seen skipped most often, in the order they tend to surface as production incidents.

1. What does the system do when the input is wrong?

A prototype assumes valid input. Production cannot.

The missing questions:

• What happens with empty input, malformed input, oversized input, or input in the wrong language?

• Is invalid input rejected at the boundary, or does it flow downstream and fail somewhere less visible?

• Is the rejection typed and recorded, or does it surface as a generic error?

• Can a malformed input cause a partial write, a silent data corruption, or a retry storm?

A system that fails on bad input is not necessarily broken. A system that fails invisibly on bad input is.

2. What does the system do when a dependency fails?

A prototype runs when its dependencies are healthy. Production runs when they are not.

The missing questions:

• What happens when the model API times out after the request has already been partially processed?

• What happens when a retry succeeds on the second attempt but the first attempt already wrote a row?

• Is there a timeout budget, and does it account for the slowest dependency in the chain?

• Is there a fallback path, and has it been tested under the same conditions that trigger it?

The hardest dependency failures are the ones that happen after a successful write. They are the ones that produce duplicates, orphaned records, and reconciliation work that nobody planned for.

3. What does the system do when it is slow?

A prototype has no latency requirement. Production does.

The missing questions:

• What is the p50, p90, and p99 latency, and which of those does the user experience?

• What happens to the request queue when latency increases?

• Does a slow response trigger a client timeout, and does that timeout trigger a retry, and does that retry increase load on the already-slow dependency?

• Is there backpressure, or does the system accept work it cannot complete?

A system that is correct but slow under load is a system that fails differently, not a system that succeeds.

4. What does the system do when it is wrong?

A prototype is evaluated on whether it produces the right answer. Production has to handle the case where it produces the wrong one.

The missing questions:

• How does the system detect that its output is wrong, incomplete, or low-confidence?

• Is there a human review step, and is it designed or improvised?

• Can a wrong output be corrected, and does the correction propagate to everything that depends on it?

• Is there an audit trail that connects a stored result to the input and configuration that produced it?

The difference between a system that fails visibly and one that stores the failure is a validation layer and a record. Neither is present in a prototype.

5. Who operates this at 3 a.m.?

A prototype has an author. Production has an on-call rotation.

The missing questions:

• Who is responsible when the system fails outside business hours?

• Is there a runbook, and has anyone followed it under pressure?

• Are the logs and metrics sufficient to diagnose a failure without the original author?

• Is there a rollback path, and has it been tested?

• Is the failure mode documented, or does it have to be rediscovered?

Operability is not a feature. It is a property of the system's design. A system that cannot be operated by someone other than its author is not a production system.

6. What does this cost at scale?

A prototype is cheap because it runs once. Production is expensive because it runs continuously.

The missing questions:

• What is the cost per request, per user, per day, and per month?

• Which cost grows linearly with usage, and which grows faster?

• Is there a cost ceiling, and what happens when it is reached?

• Are there usage fields, cache fields, and billing fields recorded per request?

• Is the cost model based on the visible output or on the total tokens consumed, including reasoning and retries?

Cost is a reliability property. A system that works but becomes unaffordable at scale is a system that fails at scale.

7. What happens when the model changes?

A prototype is built against one model version. Production is built against a moving target.

The missing questions:

• Is the model ID pinned to a stable version, or is it an alias that can change?

• Which parameters are sent, and which are effective?

• What happens when a deprecated parameter starts returning an error?

• Is there a compatibility test that runs before a model upgrade, or does the upgrade happen and the failures surface in production?

• Is the output schema stable across versions, and does the client reject unexpected shapes?

Model changes are not rare events. They are routine. A system that cannot survive a model version change is a system with a scheduled incident.

8. What is the evidence that this works?

A prototype is demonstrated. Production is measured.

The missing questions:

• What is the test set, and does it represent the input distribution?

• What is the sample size, and what difference can it resolve?

• Is the scorer stable, and is scorer disagreement reported?

• Is there a record of what was tested, when, against which model version, with which parameters?

• Is there a documented list of what was not tested?

A working demo is evidence of feasibility. It is not evidence of reliability. The evidence for reliability is a record that someone else can inspect, re-run, and disagree with.

For AI tool features, rules and availability, refer to their official sites.