What does schema-on-write commit a log store to that schema-on-read does not?

middlenice to knowfreq 34%

answer

When is the meaning of a line fixed
One decides before, one decides per query
Frozen mistakes versus repayable parsing
Keeping the raw line preserves your options

basics

~20 s

Schema-on-write freezes the fields, their types and what is searchable before data lands, so mistakes are permanent for records already written. Schema-on-read keeps the raw line and derives structure per query, so the interpretation stays changeable and retroactive.

solid answer

~50 s

Under schema-on-write somebody declares fields, types and what is searchable up front; the write path applies that declaration and stores the structured result. Queries are then fast and aggregatable, but the decision is frozen: a field you failed to extract cannot be recovered from data already written, two services disagreeing on a field's type collide at ingest, and you pay for structure nobody queries. Under schema-on-read the line is stored close to how it arrived and interpreted by whatever query needs it, so a parsing mistake is repairable over data you already hold - at the price of paying that parsing on every query, absorbing format drift silently, and discovering an unparseable line mid-incident. Note this is about when the decision is frozen, not about which process does the parsing. Most real platforms extract a small agreed set of fields and keep the raw line for everything else.

Under **schema-on-write** somebody declares fields, types and what is searchable up front; the write path applies that declaration and stores the structured result. Queries are then fast and aggregatable, but the decision is frozen: a field you failed to extract cannot be recovered from data already written, two services disagreeing on a field's type collide at ingest, and you pay for structure nobody queries. Under **schema-on-read** the line is stored close to how it arrived and interpreted by whatever query needs it, so a parsing mistake is repairable over data you already hold - at the price of paying that parsing on every query, absorbing format drift silently, and discovering an unparseable line mid-incident. Note this is about *when the decision is frozen*, not about which process does the parsing. Most real platforms extract a small agreed set of fields and keep the raw line for everything else.

code

json · 2 lines

json

{"ts":"2026-05-14T08:31:07.412Z","service":"booking-api","level":"WARN",
 "trace_id":"9f1c...","msg":"hold expired ferry_id=48213 seats=2 waited=734ms"}

go deeper

for a junior

Be ready to say which one decides the structure before storing and which one decides while querying, and that the second keeps the original text around so it can be re-interpreted later.

for a middle

Explain the consequences in both directions: type collisions and unrecoverable fields on one side, repeated parsing cost and silent format drift on the other.

for a senior

Argue for the hybrid deliberately - a small agreed indexed set plus the retained raw line - and describe how you police shared field names across teams so a name means one thing.

for a principal

Own it as a governance question: freezing structure means coordinating every producing team, and the amount of structure you mandate is a bet on which questions the organisation will ask.

The two commitments

Every log store decides when the meaning of a log line is fixed.

Schema-on-write fixes it before the data lands. Somebody declares the fields, their types and which of them are searchable; the write path parses each record against that declaration, and what is stored is the structured, typed, indexed result. Ambiguity is resolved once, at ingest, by the platform.

Schema-on-read defers it. The store keeps the line roughly as it arrived, tagged with a few identifying attributes, and any structure is extracted by the query that needs it. The same stored bytes can be interpreted three different ways by three different queries, and the interpretation can be changed tomorrow.

This is a different axis from where the parsing runs - both models can parse in an agent, in the store, or in the query engine. The question here is whether the decision is frozen at ingest or re-taken on every read.

What each buys

	Schema-on-write	Schema-on-read
When structure is decided	Before ingest, once	At query time, every time
Query speed on known fields	Fast; typed and pre-computed	Slower; parsing repeated per query
Aggregations and ranges	Natural, the values are typed	Possible, but computed during the scan
Cost of a new log format	Coordination before it ships	Absorbed; fix the query later
Cost per query	Low	Parsing on every matching line
Cost per stored byte	Higher; derived structures	Lower; close to compressed raw
Recovering from a mistake	Only for data written afterwards	Retroactive, over data you already hold

The asymmetry in the last row is the heart of it. Schema-on-write mistakes are permanent for the data already written; schema-on-read mistakes are repairable, because the original text is still there.

What being wrong costs

Getting schema-on-write wrong shows up as:

The field you need was never extracted. During an incident you want to group by a value that lives in the middle of a message, and no amount of cleverness recovers it from last week's records. Adding it now helps only from now on.
Type collisions across teams. Two services log a field with the same name and different shapes - one a number, one an object. Where a declared type already exists, the mismatched records or fields are rejected or coerced, and the data loss is discovered later by someone who assumed the search was complete.
Paying for structure nobody queries. Every declared, indexed field costs on the write path and on disk for the lifetime of the data, whether or not a single query has ever mentioned it.
Coordination as a tax. A team cannot ship a new log format without touching a shared declaration, so people quietly stuff information into free text to avoid the process - defeating the model entirely.

Getting schema-on-read wrong shows up as:

Parsing paid over and over. A dashboard refreshing every thirty seconds re-derives the same fields from the same lines forever; on a ferry-timetable booking platform pushing 1.7 TB of logs a day, that is real, recurring CPU.
Formats that drift under you. A service changes its message shape mid-window and the query-time extraction now has to handle both shapes, or silently returns fewer results than it should. Nothing failed at ingest to warn you.
Discovery at the worst moment. The line turns out to be unparseable - a delimiter inside a value, a truncated payload, an unescaped newline - and you find out at three in the morning with the extraction expression in front of you.
Analytics that do not scale. Aggregating across a long window means parsing everything in that window, so heavy questions are slow or capped, and people stop asking them.

What real platforms do

Most production estates sit deliberately in the middle: extract and index a small set of high-value, agreed fields - timestamp, service, level, and a correlation identifier such as a trace id - and keep the rest of the line as text, interpreted at read time. That preserves the fast, aggregatable, alertable path for the fields everyone shares, and keeps the permanent-mistake surface as small as possible.

Two rules make that middle work. First, the shared fields must be genuinely shared - a name that means different things in two services is worse than no field at all. Second, always keep the original line. A store that discards the raw text after extraction converts every future parsing improvement into data you can no longer reprocess, which is the one property that made the deferred model valuable in the first place.

## The two commitments Every log store decides *when* the meaning of a log line is fixed. **Schema-on-write** fixes it before the data lands. Somebody declares the fields, their types and which of them are searchable; the write path parses each record against that declaration, and what is stored is the structured, typed, indexed result. Ambiguity is resolved once, at ingest, by the platform. **Schema-on-read** defers it. The store keeps the line roughly as it arrived, tagged with a few identifying attributes, and any structure is extracted by the query that needs it. The same stored bytes can be interpreted three different ways by three different queries, and the interpretation can be changed tomorrow. This is a different axis from *where the parsing runs* - both models can parse in an agent, in the store, or in the query engine. The question here is whether the decision is **frozen** at ingest or **re-taken** on every read. ## What each buys | | Schema-on-write | Schema-on-read | | --- | --- | --- | | When structure is decided | Before ingest, once | At query time, every time | | Query speed on known fields | Fast; typed and pre-computed | Slower; parsing repeated per query | | Aggregations and ranges | Natural, the values are typed | Possible, but computed during the scan | | Cost of a new log format | Coordination before it ships | Absorbed; fix the query later | | Cost per query | Low | Parsing on every matching line | | Cost per stored byte | Higher; derived structures | Lower; close to compressed raw | | Recovering from a mistake | Only for data written afterwards | Retroactive, over data you already hold | The asymmetry in the last row is the heart of it. Schema-on-write mistakes are **permanent for the data already written**; schema-on-read mistakes are **repairable**, because the original text is still there. ## What being wrong costs Getting schema-on-write wrong shows up as: 1. **The field you need was never extracted.** During an incident you want to group by a value that lives in the middle of a message, and no amount of cleverness recovers it from last week's records. Adding it now helps only from now on. 2. **Type collisions across teams.** Two services log a field with the same name and different shapes - one a number, one an object. Where a declared type already exists, the mismatched records or fields are rejected or coerced, and the data loss is discovered later by someone who assumed the search was complete. 3. **Paying for structure nobody queries.** Every declared, indexed field costs on the write path and on disk for the lifetime of the data, whether or not a single query has ever mentioned it. 4. **Coordination as a tax.** A team cannot ship a new log format without touching a shared declaration, so people quietly stuff information into free text to avoid the process - defeating the model entirely. Getting schema-on-read wrong shows up as: 1. **Parsing paid over and over.** A dashboard refreshing every thirty seconds re-derives the same fields from the same lines forever; on a ferry-timetable booking platform pushing 1.7 TB of logs a day, that is real, recurring CPU. 2. **Formats that drift under you.** A service changes its message shape mid-window and the query-time extraction now has to handle both shapes, or silently returns fewer results than it should. Nothing failed at ingest to warn you. 3. **Discovery at the worst moment.** The line turns out to be unparseable - a delimiter inside a value, a truncated payload, an unescaped newline - and you find out at three in the morning with the extraction expression in front of you. 4. **Analytics that do not scale.** Aggregating across a long window means parsing everything in that window, so heavy questions are slow or capped, and people stop asking them. ## What real platforms do Most production estates sit deliberately in the middle: extract and index a small set of high-value, agreed fields - timestamp, service, level, and a correlation identifier such as a trace id - and keep the rest of the line as text, interpreted at read time. That preserves the fast, aggregatable, alertable path for the fields everyone shares, and keeps the permanent-mistake surface as small as possible. Two rules make that middle work. First, the shared fields must be genuinely shared - a name that means different things in two services is worse than no field at all. Second, always keep the original line. A store that discards the raw text after extraction converts every future parsing improvement into data you can no longer reprocess, which is the one property that made the deferred model valuable in the first place.

Two services log a field of the same name with different shapes. What happens under each model?
Under schema-on-write the declared type wins: the conflicting records or fields are rejected or coerced at ingest, and the loss is usually found much later by someone who assumed their search was complete. Under schema-on-read nothing fails at ingest; the disagreement surfaces as a query that returns fewer results than expected, and it can be fixed by changing the extraction - including for the data already stored.
Why do most production platforms sit in the middle rather than picking one?
Because a small set of shared fields - timestamp, service, level, a correlation identifier - carries nearly all the aggregation, alerting and navigation value, and it is cheap and stable enough to freeze. The rest of the message is where formats drift and predictions fail, so it stays raw text interpreted at read time. That keeps the permanently-wrong surface small while retaining fast typed queries where they matter.
What breaks if a schema-on-read store discards the original line after extraction?
The one property that made it worth choosing. Retroactive reinterpretation depends on the raw text still being there; without it, every improvement to an extraction expression applies only to future data, so you have taken on the recurring parsing cost and the permanence problem at the same time. Always keep the original record.

saying these in an interview costs you the question

Says schema-on-read means the logs have no structure
Thinks a schema change re-processes data already stored
Claims schema-on-read is free because ingest is cheap
Confuses the choice with which process performs the parsing
Discards the raw line once fields have been extracted

What does schema-on-write commit a log store to that schema-on-read does not?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The two commitments

What each buys

What being wrong costs

What real platforms do

▾common follow-ups3 questions

▾red flags5 flags

The two commitments

What each buys

What being wrong costs

What real platforms do

answer

go deeper