CAP theorem
~11 minTwo data centres hold the same data. The link between them breaks. Each must now answer from its own copy, or refuse. CAP is the proof that there is no third way.
Trains you for data engineer interviews across coding, SQL, and pipeline-design rounds. Covers Python over data stores, warehouse modeling, distributed processing, streaming, and cloud infrastructure.
10769 of 10769 questions in your plan: 0 known, 0 in progress, 0 due, 0 skipped, 10769 new. 0 cut, outside your plan.
A data engineering loop grades data before it grades code. The SQL round comes first, often as the screen, and it is live: write the query, then say what it returns when a dimension row is missing, when a key repeats on the right side of a join, and when two rows tie on the ordering a `ROW_NUMBER()` depends on. The Python round follows, and in this role a data-manipulation task against a file or an API turns up about as often as an algorithm problem. Middle-level loops and above add a modeling or pipeline-design round, the one that carries the most weight; a behavioral interview comes last. The design rounds are built from things that went wrong. A nightly job was rerun and the fact table now holds each row twice: was the load an append or an overwrite by partition? An event arrived three days after the hour it belongs to: does the aggregate get corrected, and by what? A Spark job that took ten minutes now takes two hours: which key skewed, what spilled to disk, would a broadcast join remove the shuffle or only move it? A consumer group's lag climbs while every broker looks healthy: how many partitions does the topic have, and how many consumers can that number keep busy? What separates levels is which question the candidate asks first. One candidate reaches for the tool and names the right operator or setting. Another asks how much data there is, how late it may arrive, what a wrong row costs the team that reads it and what the job spends per run, then picks the design that is cheapest to backfill and easiest to prove correct. Senior interviewers wait for that second answer, and notice when the grain of a table, the idempotency of a load or the meaning of "exactly once" is stated unprompted.
[Python](/topics/lang-python) and [SQL](/topics/lang-sql) open the plan as a pair because the first two rounds draw on them directly, and because in this role SQL is a language you write transformations in, not a tool you query a database with. The systems foundations come next, before any data tool, on the argument that a pipeline is a distributed system: [serialization](/topics/found-serialization) explains why adding a field broke a reader on the previous release, [consistency models](/topics/found-eventual-consistency) explain what a replica can promise, and the resilience patterns explain what a retry does to a load that was not idempotent. Linux, the [coding-round algorithms](/topics/found-dsa), Git with GitHub and [HTTP](/topics/proto-http), the protocol most ingestion pages through, finish the groundwork. The plan then moves through storage in the direction data flows. [Relational concepts](/topics/db-relational-concepts) are the source systems: transactions, replication, and the write-ahead log that log-based change data capture reads. Document and wide-column stores are the other shapes a source takes, and the compaction and tombstones under the wide-column store return later inside a lakehouse table. [Analytical databases](/topics/db-analytical-concepts), columnar and massively parallel, are the destination. Modeling then asks what shape the destination should hold: [dimensional models](/topics/data-modeling) and their slowly changing dimensions, then the [table formats](/topics/data-table-formats) that give files on object storage the snapshot isolation and schema evolution a warehouse table already has. Compute comes only after the data has a shape. [Processing engine concepts](/topics/data-processing-concepts) precede [Spark](/topics/data-spark) so that shuffles, skew, spill, watermarks and checkpoints have engine-neutral names before Spark gives them its own. Then scheduling and movement: [orchestration](/topics/data-orchestration) and dbt put the transformations on a schedule with dependencies, and [ingestion](/topics/data-ingestion) covers how rows arrive at all, from file drops and change data capture to product-analytics events and reverse-ETL. The plan turns from batch to continuous with [broker operations](/topics/cloud-messaging-concepts) in general, then [Kafka](/topics/cloud-kafka) as the broker most pipelines run on. Analysis and reporting closes this half from the consumer's side: how an analyst will [join and reshape a table](/topics/data-libraries-concepts) in one process, or in a BI tool, decides how you publish it. The last third is the platform the pipelines run on and the obligations attached to the data. The cloud stretch, from provider concepts through containers and Docker to [infrastructure as code](/topics/cloud-iac-concepts), comes before testing, security, delivery and observability, in roughly the order a team adopts them. [Governance](/topics/data-governance) with GDPR and CCPA sits near the end deliberately: lineage, PII masking and erasure requests only make sense once you know how a table format stores and rewrites a row, and the regulations turn those questions into deadlines. The AI-assistant module and the [behavioral round](/topics/found-behavioral) are left for last, two stretches the technical rounds do not depend on.
A backend engineer moving into data already has the Python, Git, HTTP and container modules and is usually short on window functions under time pressure, the grain and history of a dimensional model, and what a shuffle costs: study by topics from the storage modules onward and treat SQL as a daily exercise. An analyst moving the other way has SQL, modeling and the BI tools and is short on the engineering around them: the Python module to its end, the systems foundations, Linux and the cloud stretch, plus daily coding practice, because an analyst's loop rarely had that round. Two weeks out, cut the plan down with its own controls: narrow level to what the posting advertises and raise importance to must know; study order already opens at by frequency, so the SQL, Python and modeling questions that survive the cut are the ones asked most. A three-month runway is long enough to build a pipeline of your own while you study by topics from the first module down: a source table, a scheduled transformation, a modeled table someone else can query. The design round is easier to answer about a job you have rerun after it failed. Seniors lose little by skimming the language modules; their hours belong in processing concepts, streaming and governance, and in rehearsing the arithmetic a senior design answer is judged by, from bytes shuffled and minutes of lag to the price of a backfill. Spark's MLlib section is the plan's only step toward training a model. Feature stores, experiment tracking, serving and retraining when the data drifts belong to the [MLOps Engineer roadmap](/roadmaps/mlops), which shares much of this plan and adds the model's lifecycle. The warehouse appears here as concepts rather than as a product; when a posting names one, the [Snowflake](/roadmaps/snowflake) or [BigQuery](/roadmaps/bigquery) roadmap covers that engine on its own terms.
Two data centres hold the same data. The link between them breaks. Each must now answer from its own copy, or refuse. CAP is the proof that there is no third way.
A data engineering loop grades data before it grades code. The SQL round comes first, often as the screen, and it is live: write the query, then say what it returns when a dimension row is missing, when a key repeats on the right side of a join, and when two rows tie on the ordering a ROW_NUMBER() depends on. The Python round follows, and in this role a data-manipulation task against a file or an API turns up about as often as an algorithm problem. Middle-level loops and above add a modeling or pipeline-design round, the one that carries the most weight; a behavioral interview comes last.
The design rounds are built from things that went wrong. A nightly job was rerun and the fact table now holds each row twice: was the load an append or an overwrite by partition? An event arrived three days after the hour it belongs to: does the aggregate get corrected, and by what? A Spark job that took ten minutes now takes two hours: which key skewed, what spilled to disk, would a broadcast join remove the shuffle or only move it? A consumer group's lag climbs while every broker looks healthy: how many partitions does the topic have, and how many consumers can that number keep busy?
What separates levels is which question the candidate asks first. One candidate reaches for the tool and names the right operator or setting. Another asks how much data there is, how late it may arrive, what a wrong row costs the team that reads it and what the job spends per run, then picks the design that is cheapest to backfill and easiest to prove correct. Senior interviewers wait for that second answer, and notice when the grain of a table, the idempotency of a load or the meaning of "exactly once" is stated unprompted.
Python and SQL open the plan as a pair because the first two rounds draw on them directly, and because in this role SQL is a language you write transformations in, not a tool you query a database with. The systems foundations come next, before any data tool, on the argument that a pipeline is a distributed system: serialization explains why adding a field broke a reader on the previous release, consistency models explain what a replica can promise, and the resilience patterns explain what a retry does to a load that was not idempotent. Linux, the coding-round algorithms, Git with GitHub and HTTP, the protocol most ingestion pages through, finish the groundwork.
The plan then moves through storage in the direction data flows. Relational concepts are the source systems: transactions, replication, and the write-ahead log that log-based change data capture reads. Document and wide-column stores are the other shapes a source takes, and the compaction and tombstones under the wide-column store return later inside a lakehouse table. Analytical databases, columnar and massively parallel, are the destination. Modeling then asks what shape the destination should hold: dimensional models and their slowly changing dimensions, then the table formats that give files on object storage the snapshot isolation and schema evolution a warehouse table already has.
Compute comes only after the data has a shape. Processing engine concepts precede Spark so that shuffles, skew, spill, watermarks and checkpoints have engine-neutral names before Spark gives them its own. Then scheduling and movement: orchestration and dbt put the transformations on a schedule with dependencies, and ingestion covers how rows arrive at all, from file drops and change data capture to product-analytics events and reverse-ETL. The plan turns from batch to continuous with broker operations in general, then Kafka as the broker most pipelines run on. Analysis and reporting closes this half from the consumer's side: how an analyst will join and reshape a table in one process, or in a BI tool, decides how you publish it.
The last third is the platform the pipelines run on and the obligations attached to the data. The cloud stretch, from provider concepts through containers and Docker to infrastructure as code, comes before testing, security, delivery and observability, in roughly the order a team adopts them. Governance with GDPR and CCPA sits near the end deliberately: lineage, PII masking and erasure requests only make sense once you know how a table format stores and rewrites a row, and the regulations turn those questions into deadlines. The AI-assistant module and the behavioral round are left for last, two stretches the technical rounds do not depend on.
A backend engineer moving into data already has the Python, Git, HTTP and container modules and is usually short on window functions under time pressure, the grain and history of a dimensional model, and what a shuffle costs: study by topics from the storage modules onward and treat SQL as a daily exercise. An analyst moving the other way has SQL, modeling and the BI tools and is short on the engineering around them: the Python module to its end, the systems foundations, Linux and the cloud stretch, plus daily coding practice, because an analyst's loop rarely had that round.
Two weeks out, cut the plan down with its own controls: narrow level to what the posting advertises and raise importance to must know; study order already opens at by frequency, so the SQL, Python and modeling questions that survive the cut are the ones asked most. A three-month runway is long enough to build a pipeline of your own while you study by topics from the first module down: a source table, a scheduled transformation, a modeled table someone else can query. The design round is easier to answer about a job you have rerun after it failed. Seniors lose little by skimming the language modules; their hours belong in processing concepts, streaming and governance, and in rehearsing the arithmetic a senior design answer is judged by, from bytes shuffled and minutes of lag to the price of a backfill.
Spark's MLlib section is the plan's only step toward training a model. Feature stores, experiment tracking, serving and retraining when the data drifts belong to the MLOps Engineer roadmap, which shares much of this plan and adds the model's lifecycle. The warehouse appears here as concepts rather than as a product; when a posting names one, the Snowflake or BigQuery roadmap covers that engine on its own terms.