From Notebook Experiment to Repeatable Team Workflow

The question

A notebook produced a result that a team is about to depend on. What has to change before that result becomes something the team can reproduce, and who is responsible for the change?

The usual answer is “clean it up and put it in a repo,” and that answer is where most of the failures in this post begin. Cleaning up a notebook produces a tidier notebook. It does not produce a workflow, because a notebook is a record of an execution and a workflow is a specification of a process — and the gap between them is not cosmetic. The notebook records what happened when the author ran cells in a particular order, with particular state in the kernel, against particular files on a particular machine. The workflow has to specify what must happen regardless of who runs it, in what order, with what state. Replacing the first with a cleaned-up version of the first does not produce the second.

This post is about the transition: what a notebook actually captures, what it leaves out, the five stages between an exploration artifact and a service, and the workflow record that makes the transition checkable.

The short answer

The transition is not a cleanup. It is a replacement of an execution record with a process specification, followed by evidence that the specification produces a comparable result.

Most teams attempt the jump from stage 0 to stage 4 in one move and end up at a stage 1 script with stage 4 ambitions. The result is a workflow that runs, produces output, and cannot be checked — which is the prototype-to-production gap at team scale.

The rest of this post is why the jump fails, and what each stage actually requires.

What a notebook captures that a workflow does not

A notebook is a faithful record of one execution, and one execution includes state that the notebook does not display. The state is what makes the notebook work on the author’s machine and fail on anyone else’s.

Kernel state. Variables defined in cells that were run earlier, deleted, re-run, or modified after definition. A cell that references df may be relying on a df that was created eight cells ago, filtered in a cell that is now marked as a scratch note, and joined with a file that was loaded manually. The notebook shows the code. It does not show which version of df the kernel is holding.

Execution order. Notebook cells are a document with an implied order that is not the execution order. A run where cell 3 executed before cell 1, cell 1 before cell 7, and cell 5 was executed twice is a valid notebook and an invalid workflow. The output at the bottom is a function of the execution history, and the history is not in the file.

Local files. Data downloaded by hand, model weights cached at a path that only exists on one machine, a CSV that was edited in a spreadsheet between two runs. The notebook references a path. The path is not the artifact.

Secrets. An API key pasted into a cell, a credential in an environment variable set in the terminal, a .env file that is gitignored. The notebook does not say which secret was used, because the secret was never in the notebook. This is also the case where the leak risk is highest: a pasted key committed to git is a key exposed, and the notebook shows the paste.

Interactive choices. A threshold adjusted after seeing the output, a chart type selected, a prompt edited after the result looked wrong, a max_output_tokens changed between runs. These are decisions, and decisions are the most important thing a notebook fails to record — the decision-record pattern exists precisely because they do not appear in the code.

Untracked data versions. The dataset on disk is data.csv. The dataset that produced the result was a version of data.csv that has since been overwritten. The test-set freeze is the discipline that prevents this, and notebooks routinely violate it by reading from a mutable path.

None of these are notebook defects. They are properties of an execution record. The defect is treating the record as a specification.

The handoff problem

The hidden state above is recoverable if the author is available, because the author knows which state mattered. The handoff problem is that the workflow is being built precisely because the author will not always be available.

Three things follow from that:

The author’s checks live in the author’s head. A notebook that “works” is often a notebook whose failures the author noticed and worked around manually: a malformed record skipped by eye, a response that looked wrong and was regenerated by hand, an edge case that the author knew about and avoided in the prompt. The workflow has to catch those automatically, and the author is the only person who knows what they are. The handoff is the moment to ask, and most handoffs skip it because the notebook appears to work.

The author’s tolerance is not the team’s tolerance. A notebook can be run at the author’s convenience, with failures investigated when the author has time. A workflow runs on a schedule or on request, and its failures are the team’s. The tolerance for a silent failure — an empty output, a truncated record, a 200 with no usable content — drops from “I’ll check” to “this must not happen,” and the workflow has to enforce the lower tolerance.

The author’s context is not the team’s context. The author knows why the prompt says what it says, why the temperature was set to 0.2, why that one field is mapped the way it is. The workflow carries the code and not the reasons, which means the next change is made without the context and the regression is introduced without the knowledge. The reasons have to be written down, and the natural place is a short design note next to the workflow, not a comment in the notebook.

The handoff question I use: “If you were hit by a bus tomorrow, what would the team not know about this notebook?” The answers are the workflow’s requirements.

The five stages, and what each one requires

The stages are not a maturity model to be climbed for its own sake. Each stage is a claim, and the claim determines the requirements. Move to the stage the claim needs, and not further.

Stage 1 — Repeatable script. A single entry point that runs end-to-end without manual steps, from a clean checkout, on a machine that is not the author’s. This is the reproducibility floor, and it is the stage that catches the kernel state, the execution order, and the manual files. The check is simple: clone the repo into a fresh container, run one command, see whether it produces output.

What it does not require: versioned inputs, configs, validation. Those are stage 2.

Stage 2 — Team-ready run. The script takes its inputs from versioned sources, its configuration from a file that is recorded with the output, and its environment from a lockfile that pins the dependency versions. This is the stage that makes the result comparable across runs and across people, because the reproducibility checklist fields — prompt hash, dataset hash, model ID, SDK version — become properties of the workflow rather than of the author’s memory.

The specific requirements:

• Inputs. A dataset path that resolves to a versioned artifact, not a mutable local file. The dataset hash is an output of the run.

• Config. All parameters, including the ones that are easy to leave in the code (max_output_tokens, thinking_level, timeout budget, retry policy), in a file that is versioned and hashed.

• Environment. A lockfile with the SDK version, which matters because SDK behavior changes across versions and an unpinned dependency is an unrecorded variable.

• Secrets. From the environment or a secret manager, never in the repo, never in the config file, never in the record.

Stage 3 — Gated pipeline. Validation gates run on the output, the decision record is written per logical request, and a CI check runs the workflow on a small input to catch structural breakage. This is the stage where failures become visible, and it is the stage most teams skip — because the workflow appears to work, and the gates are only missed when something silent happens.

The gates are the ones this series has already documented: schema validation before the output is used, finish-reason and refusal checks before the output is accepted, and a human review step for the failure classes the automated gates cannot cover.

Stage 4 — Service. Observability, latency and cost budgets, an on-call owner, a runbook. This is where the latency and cost measurements become operational requirements rather than documentation. A workflow that runs on request does not need on-call. A workflow that runs on a schedule that users depend on does.

Most internal team workflows stop at stage 3, correctly. The error is claiming stage 4 without the operational apparatus, or claiming stage 2 while relying on stage 0 state.

What a CI check should and should not do

The CI check is the mechanism that keeps a workflow at its stage. It is also the mechanism most often built to do the wrong thing.

A CI check for a model workflow should:

• Smoke-run the pipeline on a tiny input. Three or five items, end-to-end, with the real gates. It should fail if the pipeline does not run, not if the output quality is low.

• Test the gates, not the model. A known-bad input that the schema validation should reject; a request that should trigger the timeout; a response that should be caught by the finish-reason check. The gates are the team’s code, and the gates are what CI can test reliably.

• Test the compatibility assumptions. The assumption tests — does the sampling parameter still have no effect, does thinkingBudget still fail, is the non-streaming ceiling still 32,768. These are cheap and they catch the silent layer.

• Check the cost and latency ceilings. A single-run cost above a stated threshold, or a p99 above the timeout budget, should fail the build. This is how the cost estimate stays true as the workflow changes.

• Check the environment. The lockfile hash matches; the model ID is the pinned ID, not an alias; the config file is present and valid.

A CI check should not:

• Run a full evaluation. It is too slow, too expensive, and not the CI’s job. Evaluation runs are a separate cadence, with their own sample-size and validity requirements.

• Test the model’s capability. The model is not the team’s code. A CI that fails because the model output changed is a CI that will be disabled, because model output changes legitimately and constantly. Test the workflow’s assumptions about the model, not the model.

• Be a gate on a single sample. A one-item smoke test that passes today and fails tomorrow is measuring model non-determinism, not workflow correctness. The pass condition should be “the pipeline ran and the gates were exercised,” not “the output was good.”

The distinction is the same one from the prototype post: the CI is a feasibility-and-structure instrument, not a capability instrument. It answers “does the workflow still function.” The evaluation suite answers “is the output still good,” on a different cadence, with different statistics.

The workflow record

Every run of a stage-2-or-later workflow should produce a record. The minimum:

json

{
  "workflow_id": "extraction-worker",
  "workflow_version": "git:4a1f2c9",
  "run_id": "run_01J...",
  "started_at": "2026-10-20T14:00:00-05:00",
  "finished_at": "2026-10-20T14:03:12-05:00",
  "stage": 3,
  "environment": {
    "lockfile_sha256": "...",
    "sdk": "google-genai 2.30.1",
    "python": "3.12.4"
  },
  "inputs": {
    "dataset_id": "...",
    "dataset_sha256": "...",
    "config_sha256": "...",
    "prompt_sha256": "..."
  },
  "model": {
    "model_id": "gemini-4-argon",
    "call_surface": "interactions.create",
    "thinking_level": "medium",
    "max_output_tokens": 8192
  },
  "secrets_source": "env:GEMINI_API_KEY",
  "outputs": {
    "output_location": "...",
    "output_sha256": "...",
    "records_produced": 0,
    "records_rejected": 0
  },
  "gates": {
    "schema_validation": "pass",
    "finish_reason_check": "pass",
    "refusal_check": "pass",
    "human_review_sample": "queued"
  },
  "cost": {
    "usd_total": 0.0,
    "usd_per_record": 0.0
  },
  "latency": {
    "p50_ms": 0,
    "p99_ms": 0
  },
  "failures": [],
  "not_run": ["full evaluation suite (separate cadence)"],
  "evidence_level": "runtime"
}

Five field groups carry the record’s value:

• workflow_version and environment.lockfile_sha256 together make the run reproducible. A run without a version is a result without a process.

• inputs.dataset_sha256, config_sha256, prompt_sha256 make the run comparable. The test-set freeze is not a discipline that lives in a document; it lives in these hashes.

• gates makes the run auditable. A run whose gates were not exercised is a run whose output was not checked, and the record says which.

• records_rejected is the quality signal that survives aggregation. A run that produced 1,000 records and rejected 40 is a different system from one that produced 1,040 and rejected zero, and only the rejected count distinguishes them.

• not_run states what the run did not cover. The full evaluation suite is a separate cadence, and the record says so, so the next reader does not mistake a smoke run for a full one.

The record is deliberately per-run, not per-record. The per-record decision record from the error-logging pattern is a different artifact at a different granularity, and the two are correlated by run_id. Storing both is not duplication; it is the two levels of evidence the workflow needs.

Failure cases observed

The kernel-state workflow. A notebook was “converted to a script” by copying the cells in order. The script failed on the first run, because a variable was defined in a cell that appeared later in the document but had been executed earlier. The fix added the missing definition to the top of the script, which worked — and hid the fact that the notebook’s output had depended on an execution order nobody had written down.

The hidden threshold. A prompt included a threshold tuned over several notebook runs. The threshold was in the code, but the reason for it was not, and the next engineer changed it back to what looked reasonable. The workflow’s output quality dropped by a margin nobody could attribute, because the change that caused it was a comment-adjacent constant that looked arbitrary.

The manual step. A workflow that ran end-to-end except for “upload the output to the dashboard,” which the author did by hand each week. The workflow was documented as automated. It was automated for everyone who remembered the upload step, and the week the author was on vacation, the dashboard was empty.

The unpinned environment. A workflow that ran on the author’s machine and failed in CI because the SDK version had drifted. The fix pinned the SDK in the lockfile, which had not existed. The failure was a reproducibility failure that presented as a build failure.

The credential in the notebook. An API key pasted into a cell and committed to a notebook repository. The key was discovered in a public dataset scan months later. The notebook was a research artifact that had never been intended for publication, which is the case in which notebooks are most often committed without review.

The untracked dataset. A workflow that read data/latest.csv. The dataset was updated by another team, and the workflow’s output changed with no change to the code. The run record did not exist, so the change could not be attributed, and the workflow was “fixed” by reverting a code change that had nothing to do with it.

The full evaluation in CI. A CI check that ran the full evaluation suite on every commit. It took 40 minutes and cost several dollars per run, and it failed intermittently because model output varies across runs. The team disabled it within a month. The CI that replaced it — a smoke run and a gate test — ran in 90 seconds and caught the structural failures, which was the actual job.

None of these are tooling failures. All of them are stage-mismatch failures: a workflow making a claim its artifacts do not support, and the artifacts being exactly what the notebook provided.

A workflow that checks the first five is at stage 2. A workflow that checks all twelve is at stage 3. The checklist is ordered deliberately: the early items are the ones that prevent the notebook’s hidden state from re-entering the workflow under a new name.

What this post does not prove

It does not prove that every notebook should become a workflow. Exploration is legitimate, and a notebook that answers a question and is then filed is doing its job. The error is treating the notebook that produced a decision as a workflow that can reproduce it.

It does not prove that stage 3 is the right target for every team. A low-stakes internal workflow may be correctly served by stage 2, provided it does not claim the gates it does not have.

It does not prove that CI catches everything. The CI check tests structure and assumptions. It does not test capability, and it does not catch a validation gap that no one thought to write a test for — which is the human review step’s job.

It does not resolve the “who maintains it” question. A workflow at stage 3 has an implicit owner, and the transition from notebook to workflow is also a transition from an author to an owner. That is an organizational decision the post describes but does not make.

What it does establish is the structure of the transition: a notebook records an execution, a workflow specifies a process, and the gap is the hidden state — kernel state, execution order, local files, secrets, interactive decisions, and untracked data versions. The five stages and the run record are the mechanism for closing it, and the handoff question is the mechanism for finding out what the author knows that the notebook does not say.

A working demo is evidence. It is not a production system. A notebook that ran is evidence of an execution. It is not evidence that the execution can be repeated — and the distance between those two statements is the entire transition.

Reproduction notes

• Model reference: gemini-4-argon

• Access: paid Gemini API tier

• Test date: 2026-10-20

• SDK referenced: google-genai>=2.30.1 (Python), pinned via lockfile in the stage-2 example

• Call surface referenced: interactions.create; per-request ceilings of 32,768 non-streaming direct HTTP and 65,536 streaming/synchronous Interactions

• Stage model derived from the failure patterns in prior posts: reproducibility checklist (stage 1–2), schema validation (stage 3 gate), error-logging pattern (stage 3 record), human review (stage 3 gap coverage), latency measurement and cost estimation (stage 4 budgets)

• CI scope distinction referenced from the test-set construction and prototype boundary posts: CI tests workflow structure and assumptions, evaluation tests capability, and the two run on different cadences

• Compatibility assumption tests referenced from version changes and deprecation: sampling parameters ignored, thinkingBudget rejected, MINIMAL rejected, explicit caching unsupported, pinned-ID resolution

• Workflow record schema version 1.0; per-record decision record correlated by run_id rather than duplicated

• Failure samples retained: kernel-state order dependency; hidden threshold; manual upload step; unpinned SDK; committed API key; mutable latest.csv; full evaluation in CI disabled after one month

The next and final post in this roadmap closes the operational-evidence arc with the release gate: the questions to answer before shipping an AI feature, stated as evidence requirements rather than a checklist of concerns.

For AI tool features, rules and availability, refer to their official sites.