In a triage workflow, what does a system-prompt rule telling the model never to emit parser syntax actually buy?
answer
- the pair has two ends
- lower rate, same class
- an instruction moves a distribution
- the consumer is unchanged
- mitigation measured, not resolution claimed
basics
~20 sA lower rate, not a closed class. The rule acts on a probabilistic producer, while the property that turns prose into a decision lives in the consumer, which is unchanged. Ship it as mitigation, not as a fix.
solid answer
~40 sBe precise about what changed. The rule acts on one end of the pair: it asks a probabilistic producer to avoid a shape, and it will move the rate down, often a lot, which is worth something on a workflow processing thousands of tickets. What it does not touch is the other end — the parser still assigns meaning by position to whatever string arrives, so the class is intact and the next variant that clears the wording is the same finding. It also decays quietly: the rule competes with ticket text in the same context window, and nobody is measuring it because the workflow is unattended. Report it as mitigation with a measured rate, and tell the owner plainly that the boundary lives at the consumer — different code, different owner, different cost.
go deeper
Recall that a system-prompt instruction steers a model rather than constraining it, so an output the rule forbids remains possible.
Explain why the change touches only one end: the parser still assigns meaning by position to whatever string it receives, so the class survives any variant that clears the wording.
Demonstrate the operational judgment — ship the rate reduction, measure it on a fixed corpus, record the finding as mitigated with a residual, and name the decay risk in an unattended workflow.
Own the conversation with the owner: acknowledge that the consumer-side change is somebody else's code and cost, and still refuse to let the finding be recorded as resolved on producer-side evidence.
## The conversation this question is really about A finding is filed: a customer-supplied span reached the assistant's answer and a downstream parser lifted it as a routing field. The owner's first proposal is nearly always the cheapest one available to them — add a line to the system prompt telling the model never to emit that shape. The question is not whether that is a bad idea. It is what it buys, stated honestly enough that an owner can make a decision. ## What it genuinely buys - **A real reduction in rate.** Instructions do steer generation. On a workflow that processes a large volume of tickets, moving the success rate down by an order of magnitude has value that should not be sneered at. - **Speed.** It ships today, from the team that owns the prompt, without touching a service anyone else runs. - **A measurement.** If you record the rate before and after over a fixed corpus of attempts, you now have a number, and numbers are what let the more expensive change get funded later. An engineer who dismisses all of that is being purist about a real operational tradeoff. ## What it does not buy - **The class stays open.** The construction's consequence is created at the consumer. The parser still assigns meaning by position to whatever string arrives; nothing about it has changed. Any variant that clears the wording of the rule reproduces the same finding. - **It acts on a probability, not a mechanism.** The producer is a model. An instruction shifts a distribution; it does not make an output impossible. "Never" in a prompt is a preference expressed to a system that has no way to enforce it. - **It competes with the ticket.** The rule sits in the same context window as customer-supplied text that arrives later, more specifically, and with a reason attached. That is a competition, not an override. - **It decays silently.** Prompt edits, a model change, a new summarisation step — any of these can move the rate back up, and in an unattended workflow nobody notices, because the visible artefact is a digest of outcomes that all look ordinary. ## Saying it to the owner The useful sentence is short: *telling the model not to emit that shape does not change what the consumer parses.* The finding is defined by a pair — a producer that cannot be constrained to a shape and a consumer that assigns meaning by position — and a prompt rule addresses only the first half. That is why a review that closes the ticket on the prompt change is closing it on the wrong evidence. The part that makes this a judgment rather than a lecture is cost. The producer side is a text edit by the team in the room. The consumer side is somebody else's service, somebody else's release train, and possibly a contract about what that service accepts. It is entirely reasonable to ship the prompt rule now. It is not reasonable to record the finding as resolved, because the resolution state is what determines whether anyone comes back to it. ## How to leave the finding State three things and let the owner choose: 1. **The measured effect** — attempts, successes, before and after, on the same corpus. 2. **The residual** — the class is open; the property lives at the consumer, which is unchanged. 3. **The decay risk** — no signal exists today that would tell anyone the rate moved back up, because the workflow is unattended and the digest shows outcomes rather than answers. And resist one temptation in particular: describing the prompt rule as defence in depth. Depth means independent layers that fail independently. Here both the classifier and the prompt rule act on the producer's text, and neither has a notion of the consumer's grammar — two measurements of the same wrong property is not a second layer.
- Is the prompt rule fairly described as defence in depth alongside the output classifier?No. Depth means layers that fail independently. The classifier scores harm categories in the producer's text and the prompt rule steers the producer's text; both act on the same end of the pair and neither has any notion of the consumer's grammar. Two measurements of the same wrong property is one layer described twice, and calling it depth overstates coverage to whoever reads the report.
- The owner asks for a number: how confident are you the rate went down?Only as confident as the corpus you measured on. Run a fixed set of attempts before and after and report attempts, successes and the interval — a probabilistic producer means small samples move a lot. And say plainly that the number describes today's model and today's prompt; it is not a property of the workflow, so it carries no guarantee past the next change on either side.
- How should the finding be recorded if the prompt rule ships and nothing else does?As mitigated with a measured residual rate, not closed. The distinction is not bureaucratic: a closed finding leaves the queue and stops being anyone's, while a mitigated one with a stated residual keeps the consumer-side question alive for whoever eventually owns the budget. Include what signal would reveal the rate moving back up — and be honest that today there is none.
saying these in an interview costs you the question
- Calls the prompt rule a fix rather than a rate reduction
- Treats a prompt instruction as an enforceable constraint
- Describes prompt rule plus classifier as defence in depth
- Closes the finding on evidence from the producer side only
- Dismisses the prompt change as worthless despite a real rate drop