Calling the real generative step is slow, costly and sometimes fails for reasons outside your team - what seam policy do you set?

## The decision, stated honestly The real call is the only seam that exercises the real behaviour of a product's generative step. It is also slow, it costs money on every pipeline run, and it goes red for reasons the team cannot fix: the provider refusing calls for volume, a bad network, a step that is simply slower today. Both of those facts are true at once, and a lead has to choose an arrangement rather than wish the tension away. Two positions are defensible. **Position A - no real calls in the merge gate.** Every case that can block a merge uses a deterministic seam. Real calls live in a separate, named set that runs on a schedule and before a release, and its failures go to an owner rather than to whoever pushed last. The argument: a gate that goes red for reasons outside the team's control stops meaning anything within about two weeks, and a pipeline nobody believes is worse than no pipeline. **Position B - a thin live set inside the gate.** A handful of live cases block a merge, because a suite with no live case at all can be entirely green while the step is unreachable, misconfigured, or refusing every request. That is a total product outage a deterministic suite is structurally incapable of noticing. The argument: some floor of reality has to be enforced where it is still cheap to act on, and a team that never feels the real call stops designing for it. ## What picks between them - **How often the live set goes red for outside reasons.** This is a measurable rate, not a feeling. Above a few percent of runs, Position B corrodes the gate. - **What the product does when the step is unavailable.** A feature with a well-exercised alternative path can afford to learn about an outage from monitoring; one with no alternative path cannot. - **Cost per pipeline run multiplied by runs per day.** A number, not an instinct. Some teams run the pipeline forty times a day. - **How fast the step changes underneath.** A step whose behaviour moves monthly needs its real calls more often and closer to the release decision. - **Whether a deployed environment exists** to run the scheduled set against. If it does not, the pipeline is the only place a real call can happen, and the arrangement is forced. - **The blast radius of shipping the break.** An internal tool and a customer-facing flow do not deserve the same ceremony. ## Where most teams land | Tier | Seam | Blocks a merge | Runs when | |---|---|---|---| | Most of the suite | Authored answers | Yes | Every pipeline run | | One shaped path per feature | Replayed recording | Yes | Every pipeline run | | The live set | Real calls, full checks | No | On a schedule, and before a release | | Reachability check | One real call, minimal assertion | Yes | Every pipeline run | The fourth row is the compromise that gives both arguments most of what they want. A single live case that asserts almost nothing - the step was reachable and returned something of the expected shape - costs one call, almost never goes red for content reasons, and closes precisely the hole Position B worries about. The rich live cases stay out of the gate. ## The evidence the arrangement needs A policy is not a paragraph in a document. It is a paragraph plus the numbers that justify it, kept current. 1. **The live set's rate of red results caused outside the team.** This is the number that decides whether it may block a merge, and it has to be measured rather than asserted. 2. **Cost per pipeline run and runs per day**, so the bill is a figure a lead can trade against everything else. 3. **Proof that the deterministic seams still reach every branch of the team's handling.** Otherwise "the gate is deterministic" is hiding a gate that tests very little. 4. **A maximum age between the last green live run and a release.** The live set is allowed out of the merge gate only if it is inside the release decision. 5. **A named owner and a response time for the scheduled set.** An unowned scheduled set is decoration: red for a month, and nobody surprised. 6. **A review date.** Every input above moves - the price, the stability, the release cadence - so the policy is reviewed on a schedule rather than when it finally hurts. ## How the arrangement fails It fails when the live set is quietly made green: assertions loosened until nothing can fail them, or the set skipped whenever the pipeline is busy. Both keep the dashboard tidy and destroy the only evidence the whole arrangement was resting on. If the live set is not worth reading, delete it and say so. An honest Position A is defensible; a decorative live set is not.