What error context must a quarantine table carry alongside each rejected record?

## Why the raw payload, verbatim The single most common mistake is storing the *parsed* version of a record that failed parsing. If a date column blew up on the string `31/02/2026`, a quarantine table typed `DATE` cannot hold it; if the pipeline coerced it to null before writing, the reject row now says the value was missing when in fact it was malformed. Store the record as it arrived — the raw line, the raw JSON document, the serialized bytes — in a text, JSON or binary column. That copy is the only artefact that lets a human reproduce the failure and the only input a replay can legitimately use. Where the record arrived as structured input, storing both the raw form and a best-effort parsed form is fine, but the raw form is the one that must never be lossy. ## Naming the failure so it can be grouped A free-text error message is enough to read one row and useless for operating a pipeline. What you need is a **stable reason code** — a short identifier like `null_order_id`, `amount_negative`, `unknown_currency`, `json_parse_error` — plus, separately, the human-readable message and the identifier of the rule or expectation that produced it. Codes matter because everything operational is an aggregate: reject rate *by reason*, the alert that fires when a new reason appears for the first time, the replay that selects only the rows whose reason matches the fix you just shipped. If your reject reasons are exception strings containing row values, none of that grouping works and cardinality explodes. Where a record fails several checks, decide deliberately: record only the first failure and you lose the rest; record all of them (an array of codes) and triage sees the full picture at the cost of a slightly richer schema. The richer version is usually worth it. ## The source locator "Where did this come from" has a different answer per ingestion style, and the quarantine schema should carry whichever applies: - **File-based batch**: source file name (including its dated prefix), line number or byte offset, and the batch or manifest id. - **Streaming**: the topic, partition and offset, plus the message key. - **Database extraction**: the source table and the primary key, plus the watermark value the extract was reading at. The test for whether the locator is good enough is blunt: could an engineer, given only this quarantine row, go back to the source and find the original? If not, the record's provenance is broken and you cannot prove to a producer that their system emitted it. ## Run and time metadata Record the **pipeline run id**, the **code or config version** that processed it, the **arrival timestamp** and the **rejection timestamp**. This is what turns quarantine into a diagnostic tool rather than a pile: rejects that all start at one run id point at a deployment; rejects that all start at one arrival timestamp point at a source change; rejects spread evenly across runs are ordinary data noise. Keep the record's own business or event timestamp too where it can be extracted, because replay ordering may depend on it. ## Lifecycle fields A quarantine record has a life: it arrives, it is triaged, and it either replays successfully or is closed as unloadable. Model that explicitly with a `status` column (`new`, `triaged`, `pending_replay`, `replayed`, `discarded`), an `attempt_count`, a `replay_run_id` and a `resolved_at`/`resolved_by`. Two things fall out for free: you can cap attempts so a record that keeps failing does not cycle forever, and you can alert on the age of the oldest record still in `new`, which is the metric that catches a quarantine store nobody is reading. ## What you should think twice about storing Rejected records are often the *most* sensitive rows you hold, precisely because they escaped the pipeline's normal masking, typing and access controls. A quarantine store frequently ends up with raw customer payloads sitting in a table with looser permissions than the target. Treat it as production data: same access control, an explicit retention period after which resolved records are purged, and inclusion in whatever deletion-request process applies to the target. If the raw payload cannot be retained under policy, store a redacted copy plus enough structure to diagnose, and accept that replay may then require re-extraction. ## The shape that works One wide table per pipeline (or one per source family) with the payload as JSON or text, the reason code and rule name, the locator columns, the run and time metadata, and the lifecycle columns. Partition it by rejection date so retention is a partition drop, and index it by reason code and status, because those are the two predicates every triage query and every replay query uses.