What are the default, committed, buffered and pending streams in the BigQuery Storage Write API?

## Why there are four The Storage Write API unified BigQuery's batch and streaming ingestion behind one gRPC service. To cover both, it had to expose different *visibility* and *guarantee* models over the same append mechanism. Each stream type is one point in that space. The common machinery is the same everywhere: you open a connection, send `AppendRows` requests carrying rows serialised as protocol buffers, and the service acknowledges each request. What differs is when a reader of the destination table can see those rows, and whether a retried append can duplicate them. ## The default stream Every table has an implicit stream named `_default`. You do not create or finalize it, many writers can use it at once, and rows are queryable essentially as soon as they are acknowledged. It does **not** accept offsets, and therefore gives **at-least-once** semantics: if your client retries an append whose acknowledgement was lost, the rows land twice. This is the right default for high-volume telemetry where a small duplicate rate is tolerable or is cleaned up downstream, and it is the simplest thing to operate — no stream lifecycle, no offset bookkeeping. ## Committed streams You create one with `CreateWriteStream(type=COMMITTED)`. Rows are visible to readers as soon as they are committed to the stream, so freshness matches the default stream. The difference is that each `AppendRows` request may carry an **offset** — the row position this batch starts at. The service tracks the stream's current offset and rejects an append whose offset has already been written, so a retry after an ambiguous failure is a no-op rather than a duplicate. That is how **exactly-once** streaming is achieved. The price is bookkeeping: one stream per writer, offsets tracked by the writer, and `FinalizeWriteStream` when the writer is done. Throughput per stream is also lower than piling everything into the shared default stream, so exactly-once is a real cost, not a free flag. ## Buffered streams A buffered stream accepts appends that are **not yet visible**. Visibility advances only when you call `FlushRows` with an offset — everything up to that offset becomes readable. This gives row-level control over when consumers see data, which is what connectors and frameworks want when they need to align BigQuery visibility with their own checkpointing. Most application code never needs it; it exists for integrations. ## Pending streams A pending stream buffers all appended rows invisibly. Nothing is readable until you `FinalizeWriteStream` and then `BatchCommitWriteStreams` — one call that can commit **several** streams together, atomically. At that moment every row from every committed stream becomes visible at once. That is exactly the semantics of a load job, achieved without writing files: parallel workers each own a pending stream, all of them finish, one commit publishes the whole batch, and a failure anywhere means nothing was published. It is the right choice when the data is already in your process (so writing it to Cloud Storage first would be pure overhead) but you still want batch atomicity. ## Choosing - Tolerate duplicates, want the least machinery and the most throughput → **default stream**, plus downstream deduplication if needed. - Must not duplicate, need rows visible continuously → **committed** stream with offsets. - Need to control visibility at a checkpoint boundary from a framework → **buffered**. - Want a batch to appear all at once or not at all → **pending**. ## What is the same across all four Billing is by bytes ingested in every case — stream type changes semantics, not cost. Rows land first in a write-optimised buffer and are reorganised into columnar storage in the background; queries read both, so freshness is not affected by that conversion. Schema is negotiated per connection from the protocol-buffer descriptor you supply, so a schema change on the table generally means reopening the connection with an updated descriptor. ## The legacy path for contrast Before the Storage Write API, streaming meant the REST method `tabledata.insertAll`, which sends JSON rows and offers only **best-effort** deduplication via an `insertId` per row within a short window. There is no exactly-once mode, no pending semantics and no atomic commit. New work should use the Storage Write API; recognising `insertAll` in an existing pipeline and knowing why it is being replaced is the interview-relevant part.