skip to content

Why log a prompt template's id and version beside the rendered text, and why must an eval render through production code?

level: seniorimportance: should knowfreq 22%

answer

  1. which source, and what was read
  2. rendered strings never group
  3. what a version does not cover
  4. a pasted prompt skips the logic
  5. a snapshot with no model call

basics

~20 s

A prompt template's id and version group calls by source and let you compare changes; the rendered text shows what the model actually read. An eval must call the production render, or it scores strings production never sends.

solid answer

~50 s

Each half answers a different question. The **template id and version** say which source produced a call, so you can group calls, compare before and after a change, and roll back; rendered strings can't do that because the values make nearly every one different. The **rendered text** says what the model read, which is where a render bug becomes visible, such as a dropped section or a doubled pass. A template version doesn't identify the render code, so I'd log its build too. The eval has to call the production render function with fixture inputs, because a hand-written prompt skips the optional-section, escape and language logic, and measures a string production may never send. Rendered text also holds customer data, so it commonly goes to a restricted store with its own retention, with ids and a hash in ordinary logs.

code

json · 10 lines
json
{
  "template_id": "support-reply",
  "template_version": "a41f09c",
  "render_build": "2026.11.2",
  "language": "es",
  "optional_sections": ["customer"],
  "model": "<model identifier>",
  "rendered_sha256": "9d2c...e71a",
  "rendered_ref": "restricted-store://prompts/2f6b81"
}

go deeper

for a junior

Remember that a good log entry says both which template revision ran and what text the model received. One without the other leaves questions you cannot answer later.

for a middle

Explain why you can group on a template version but not on rendered text, and why an eval that pastes its own prompts is measuring something production may never send.

for a senior

Walk an incident from a complaint to a revision to one rendered prompt. Include what a version misses, namely the render code, and how you would detect drift by comparing a re-render's hash.

for a principal

Set the logging standard with the privacy cost in view: which store holds customer text, for how long, and who can read it, while keeping one bad reply traceable to one source revision.

## What each logged field answers A call from a **prompt template** has several causes, and no single field identifies them all. Take a support-reply generator that renders one template per request: | Field | Question it answers | Why the others can't | |---|---|---| | Template id | Which prompt was this? | Rendered text differs on nearly every request | | Template version | Which revision of the source? | The id stays fixed across edits | | Render build | Which render code ran? | A library upgrade can change output with no template edit | | Input values | What filled the gaps? | A version says nothing about one request | | Rendered text | What did the model read? | It is the outcome of all the fields above | | Model and settings | What processed it? | Outputs can shift with no prompt change at all | The version can be a commit id or a hash of the template's source. A hash has a useful property: it stays stable across requests, and it changes only when the source changes. A hash of the rendered text does the opposite, changing with almost every request, which is why you can't group on it. ## An incident, step by step Suppose complaints arrive about replies opening with `Hi ,` and no name: 1. **Filter by template id** to collect the calls from this prompt. 2. **Group by version** to see whether the bad replies begin at a particular revision. 3. **Read the rendered text** of one failing call. It shows `Customer:` with nothing after it, which the template file could not have told you. 4. **Re-render** the same inputs with the same version on today's code, and compare hashes. A mismatch points to render code that has moved since, and either way the reconstruction is not what the model read. Without the version, step 2 becomes guesswork about deploy times. Without the rendered text, step 3 becomes a reconstruction, which only holds if the render is deterministic and old revisions are still available. ## Why the eval must use the production render An eval is a measurement of a string. If its fixtures are hand-pasted prompts, or if it has its own small fill function, it can measure different strings from production. Common things it then skips: - **Optional sections.** A pasted fixture always has the customer line, so the empty-name shape is never scored. - **Language selection.** The wrong section for a language never shows up. - **Escapes and passes.** A value holding placeholder syntax never goes through the real pipeline. - **Whitespace and joining.** Small differences accumulate between the fixture and the real output. The fix is structural: the eval calls the same function production calls, with fixture inputs rather than fixture prompts, and it stores what was rendered beside the score. ```pseudocode for fixture in fixtures: prompt = render_reply_prompt(fixture.inputs) # the production function assert_matches_snapshot(fixture.id, prompt) # no model call needed reply = call_model(prompt) record(fixture.id, template_version, prompt, score(reply, fixture.expected)) ``` The snapshot line is worth having by itself. A render-only test is cheap and exact: it fails on any byte change, while an eval score is noisy and costs model calls. Good fixtures for it are an empty name, a value containing literal placeholder syntax, and one request per supported language. What a suite pins in total, and how prompt files are versioned, belong with evaluation and iteration. ## The cost of logging rendered text The rendered text contains `customer_name` and `ticket_body`, so it is customer data and needs treating that way. Common practice looks like this: - **Separate stores.** Ids, versions and a hash of the rendered text go in ordinary logs. The text itself goes in a store with restricted access and its own retention. - **Check before sampling.** Sampling the text saves storage and loses rare failures, so keep it at least for calls that failed or were flagged. - **Redact with care.** Redacting before storage protects customers, but the stored string is then no longer exactly what the model read, so note that it was redacted. The figures for retention and sampling are a team's decision, and privacy obligations vary, so those are questions to settle with whoever owns them. Even so, the record has to let an engineer connect one bad reply to one source revision and one exact input.

  • Is a prompt template's version enough to reproduce a past prompt?
    Only with the values and render code too. The version fixes the source; the values came from the request, and a render library upgrade can change whitespace or escape behaviour with no template edit. Keep the render build and inputs if you need replay, and compare a hash of the re-render with the logged one.
  • Why add a render-only test if the eval already calls the model?
    It is cheap and exact. An eval score is noisy and costs model calls, so a small render change can slip through it. A snapshot of rendered text for fixtures such as an empty name, literal placeholder syntax and each language fails on any byte change, and a reviewer can read the diff directly.

  • Logging the rendered prompt alone is enough, since the template revision can be worked out from it.
  • Logging the template version alone is enough, since the rendered prompt can always be rebuilt.
  • An eval can use hand-written prompts, since they say the same thing as the template.
  • A template version fully identifies the prompt, even after the render code changes.
  • Rendered prompts are just configuration, so ordinary application logs are the right place for them.