How do you choose among Pub/Sub BigQuery subscriptions, Dataflow, direct Storage Write API and the Data Transfer Service?

## Frame it as ownership, not features Every ingestion path into BigQuery ends in the same two places — a load job or the Storage Write API. What differs is who writes the retry logic, who runs the process, and who gets paged. That is the axis a principal-level answer is graded on. ## Pub/Sub BigQuery subscription Pub/Sub can deliver messages **directly** into a BigQuery table by writing through the Storage Write API on your behalf. There is no pipeline to deploy and nothing to run: you configure the subscription with the destination table and either write the whole message into a column or map the topic's schema onto table columns. Use it when the message is already table-shaped and needs nothing done to it. Its limitation is exactly its virtue — there is no place to put transformation logic. Messages that fail to conform go to the subscription's dead-letter topic; anything richer than that belongs elsewhere. ## Dataflow Dataflow (managed Apache Beam) is the choice when rows need real processing on the way in: parsing and validating, joining against a reference source, windowed aggregation, deduplication, fan-out to several sinks, or a custom dead-letter path. It writes to BigQuery via the Storage Write API, and Google publishes templates for the common shapes so "Dataflow" does not automatically mean writing Beam code. The cost is a running service with its own scaling, cost and failure modes. A team that adopts Dataflow to reformat one field has bought an operational surface it did not need. ## Direct Storage Write API from your own service When the rows originate inside a service you already run, calling `AppendRows` from that service removes an entire hop — no queue, no pipeline, lowest latency. In return you own the client: connection management, retries, backpressure when BigQuery pushes back, stream lifecycle if you need exactly-once via offsets, and schema evolution when the table changes shape. This is a fine choice for a mature team and a poor one for a service that must not block on an analytics sink. Buffer or decouple through a queue if losing analytics writes could stall the request path. ## BigQuery Data Transfer Service DTS is a **managed, scheduled pull**. It moves data on a schedule from Google's own SaaS sources (Google Ads, YouTube, Campaign Manager and similar), from Cloud Storage, and from other warehouses and object stores for migrations. It gives you run history, retries and backfills without any code. What it is not is a transformation engine: it lands data on a schedule in the shape the source provides. Reshaping happens afterwards in SQL, which is often the right split — land raw, transform in the warehouse. ## The batch option you should keep considering None of the above is required if the consumer needs minutes rather than seconds. Landing files in Cloud Storage and running scheduled load jobs remains the cheapest path — no per-byte ingestion charge, atomic commits, trivially idempotent reruns. A large share of pipelines built on streaming would have been better and cheaper as micro-batches; establishing the true freshness requirement is the first move, not the last. For change data capture out of operational databases, Google's Datastream is the managed option that lands changes in BigQuery, again with no pipeline code to own. ## A decision sequence 1. **Freshness SLA from the consumer.** Minutes → scheduled batch loads. Seconds → continue. 2. **Transformation required?** No → Pub/Sub BigQuery subscription. Yes → Dataflow. 3. **Do we already own the producing process, and is analytics ingestion safe to couple to it?** Yes → Storage Write API directly. 4. **Is the source someone else's system on a schedule?** → Data Transfer Service. 5. **Duplication tolerance** then selects the stream type underneath whichever path you chose. ## What separates a strong answer Naming the products is the easy half. The judgment is in refusing to let "real-time" be assumed, in counting the operational surfaces the team will have to run at 3am, and in noticing that a managed pass-through path removes a whole class of incidents that a hand-written client will eventually produce. State the assumption you are making about the freshness requirement out loud — that is the variable the whole decision hangs on.