Prompt engineering

Read the wrong outputs, group them by hand into a few named failure modes, and count each. The biggest group sets the next revision's target, and each mode implies a different edit — or tells you the fix is not a prompt edit at all.

The label strings appearing in the demonstrations act as the effective output vocabulary. A class named only in the instruction but never demonstrated is emitted rarely or never, so every class you want back must appear at least once, spelled exactly as your parser expects.

Treat the exemplar pool as a versioned production dataset, not a folder of examples. Give every entry a timestamp and a validity window, filter expired entries out at retrieval, verify anything written back before it becomes teachable, and audit the pool against the current taxonomy on a schedule.

Majority-label bias: the model reads the label mix in the demonstrations as evidence about how often each outcome occurs, so a 3:1 approve-heavy shot set inflates the predicted approve rate on borderline building-permit applications, independent of what each application actually says.

Positive-only demonstrations teach where the rule applies but never where it stops, so the model over-generalizes and flags look-alikes. Near-miss negatives — cases that resemble a hit but are not one — draw the boundary the positives leave undefined.

Handle it at interpolation time, before the prompt is built: pick a delimiter the content cannot contain, escape or normalise the offending marker in the span, or extend the fence beyond any run inside. Never concatenate raw content and hope.

Adherence decays over a long generation: the format spec recedes into distant context, each new row is copied from nearby rows rather than the schema, and one omission propagates. Fix by batching, per-row validation with row-count postconditions, and constrained decoding.

Nothing in the serving stack checks compliance. Every layer arrives as tokens in one context window, separated only by role markers the model was post-trained to weight. That produces a strong statistical preference that can be argued down, not a rule the runtime enforces.

Replay fixed conversation scripts and assert the same constraint at several depths — say turn 5, 20 and 50 — instead of testing one-shot. Run each script repeatedly because sampling is nondeterministic, and report an adherence curve per depth so a regression shows up as decay, not a single failure.

Most of the trigger phrasing is. Models that deliberate internally already produce a chain, and long hand-written reasoning scaffolds can cut across it. What still pays is task-specific direction — what to check, in what order, and what the answer must contain.

Treat each prompt as code: a reviewed file under version control, an identifier recorded with every production response, an eval run attached to each change, a named owner, and a rollback that is one deploy away.

Only after a second, untouched split confirms the margin. Picking the best of twenty candidates inflates the winner's dev score by selection noise, and orders tuned on one model version frequently stop being best on the next — so the ongoing revalidation cost is part of the decision.

Fix one skeleton — role, rules, tools, data, task — and make every prompt fill its slots rather than invent an order. The gain is reviewability, clean diffs, eval attribution and a stable shared prefix; the cost is ceremony, so keep a documented escape hatch.

Treat it as an experiment, not a preference. Run both variants over a labelled set, compare quality, p95 latency, and cost per request including retries, then price the observability and routing you give up. Collapse adjacent stages whose boundary carries no check.

When the schema makes truth unrepresentable or removes reasoning room. Required non-nullable fields push the model to fabricate, a closed enum forces a wrong nearest label, and constraining from the first token denies it space to think before committing.

Platform policy is a floor tenants may narrow but never widen. Assemble the prompt server-side so tenant text sits in a lower layer it cannot edit or precede, define the default resolution as failing toward the platform rule, and enforce anything consequential in tools rather than prose.

Split standing constraints by blast radius. Persona and tone slip is a quality signal with a tolerable decay budget, managed by measurement and light re-anchoring. Rules whose violation is an incident — prohibitions, disclosures, machine-parsed formats — must not depend on prompt adherence at all; enforce them outside the model.

Skeleton-of-thought first asks for a terse outline, then expands each outline point in its own call, running the expansions concurrently and stitching the results. It cuts wall-clock time for long outputs but breaks down when the sections depend on each other.

An exemplar pool is policy encoded as data, so it needs a named owner in the domain team, provenance and PII review on every entry, versioning alongside the prompt, and a refresh triggered by policy changes, label-set changes and drift — not by the calendar alone.