How would you govern Langfuse prompt changes that ship without a deploy?

## The thing you actually gave away Moving prompts out of the codebase removes them from every control your organisation built around code: review, CI, staged rollout, change log, rollback procedure, blameless correlation with deploys. That is not an argument against prompt management — the iteration speed is real and product people editing prompts directly is often the whole point. It is an argument for rebuilding the specific controls you still need, and consciously not rebuilding the ones you do not. ## Decide the blast radius per surface The first decision is not a process, it is a taxonomy. Not every prompt deserves the same regime: - **Low stakes, high iteration** — marketing copy generation, internal summarisation. Follow a label, let a product owner move it, review after the fact. - **Customer-facing, recoverable** — support drafting, search rephrasing. Follow a label, but gate promotion on an offline run and restrict who may promote. - **Safety, legal or compliance relevant** — anything asserting policy, handling regulated advice, or constraining what the model may say. Pin `version=` and ship changes with a deploy. You are deliberately buying back the slowness. A single organisation-wide policy either strangles the fast cases or under-protects the dangerous ones. ## The controls worth having **Permissions.** Separate the right to create a version from the right to move `production`. Creation should be open; promotion should not. This is the single highest-leverage control because it re-introduces a human decision point at the release boundary. **A promotion gate.** Require the candidate version to have been run against a held-out dataset and its results reviewed before the label moves. The gate does not need to be automated to be effective, but automating it — a job that runs the candidate over a dataset and posts the comparison — is what makes it survive contact with a deadline. **Version attribution on every generation.** Without the prompt version recorded on the generations it produced, a promotion is an invisible change and a regression is unattributable. This is the observability precondition for everything else in this list. **Config travelling with the text.** Keep the model, temperature and any structural settings in the prompt's `config`, so a version rewritten for a different model cannot half-apply — you cannot end up with the new text and the old model. **A rollback drill.** Relabelling is fast, but people need to have done it once before an incident. Also set the expectation explicitly: because each process caches prompts, rollback converges across the fleet over roughly a cache TTL, so "I rolled back" means "it will stop within a minute", not "it has stopped". **Fallback hygiene.** The fallback literal in your code is a prompt that nobody reviews, nobody evaluates and nobody updates — and it serves real traffic during exactly the moments things are already going wrong. Sync it to the last promoted text on each deploy, or accept that your degraded mode is untested. ## Rollout is gradual, and that cuts both ways A label move is atomic on the server and gradual in your fleet, because each process picks it up when its cache entry expires. Nobody designed this as a canary and you should not pretend it is one — you cannot control the split or hold it. But it does mean a catastrophically bad prompt is not instantly everywhere, and it means any comparison of two versions over a time window is comparing unrandomised slices of traffic. Treat live per-version numbers as a signal to investigate, not as an experiment result. ## What to say when asked how far to go The honest principal answer is that the governance should be proportional to the reversal cost, not to the prompt's visibility. A prompt that produces text a human reviews before sending can be governed loosely. A prompt whose output is auto-sent, auto-executed, or legally binding needs to be pinned to a version and changed through the same pipeline as code, because the ability to change it in ten seconds is a liability rather than a feature. Most organisations get this backwards: they apply heavy process to the prompts that are easy to fix and none to the ones that are not. And there is a cultural point worth naming. Once product owners can edit prompts, prompt changes stop appearing in engineering's change log entirely. Whatever you build, make sure a promotion is visible somewhere an engineer investigating an incident will look — a notification, an annotation on a dashboard, an entry alongside deploys. An invisible release is the actual failure mode, not a bad prompt.