When Great Expectations validates a billion-row warehouse table, where does the computation run?

## The question behind the question "Does Great Expectations pull my table into Python?" is really a question about pushdown, and the honest answer is: it depends entirely on which backend the data source uses. Candidates who answer with a flat yes or no have usually only ever run it on a CSV. ## The three execution paths **SQL backend.** When the data asset is a table or query in a database or warehouse, expectations are translated into SQL. `expect_column_values_to_not_be_null(column="order_id")` becomes an aggregate that counts rows and counts nulls; `expect_column_values_to_be_unique` becomes a grouped count comparison; `expect_column_values_to_be_between` becomes a filtered count. The warehouse does the scanning and returns a handful of numbers plus, if you asked for samples, a small set of offending values. Memory in the Python process stays trivial. What is not trivial is warehouse spend: a suite of thirty expectations can issue a lot of scans over the same table, and on a columnar warehouse billed by scanned bytes or compute time that is a real line item. **Pandas backend.** When the asset is a file - a CSV, a Parquet drop - or a dataframe you hand in at run time, the batch lives in the Python process. Evaluation is in-memory and fast for modest data, and it will happily attempt to load something far larger than the container's memory limit if you point it at one. This is the correct backend for validating a landed file *at the size files actually land*, and the wrong one for a warehouse table of any size. **Spark backend.** The batch is a Spark dataframe and expectations become distributed operations. This is the path for lake-scale data already being processed by Spark, and it inherits Spark's own economics - cluster startup, shuffle costs on the expensive expectations. ## Batch definition is the real lever Regardless of engine, the biggest cost decision is *what you validate*, not how. A batch is a defined slice - one day's partition, one landed file, one query result. Validating the full history of a fact table nightly is almost always waste: yesterday's rows have not changed, and re-scanning them buys nothing but a bill. Define the batch as the increment the pipeline just produced and let the expectations describe that increment. There are two honest exceptions. Uniqueness of a key is a whole-table property - checking uniqueness within today's partition does not prove the key is unique against history, and if that global property matters you must either scan wider or enforce it in the table itself. And a distribution or reconciliation expectation may legitimately need a longer window. Make those choices explicitly rather than defaulting to the full table for everything. ## Which expectations cost the most On a SQL backend, not all expectations are priced alike. Null counts and range filters are cheap single-pass aggregates. Uniqueness requires a grouping over the column. Distinct-value expectations over a high-cardinality column can be expensive. Anything that has to sort or materialize distinct sets is where a suite's cost concentrates - worth knowing when someone asks why the quality job's warehouse usage doubled. ## The result payload trap `result_format` controls how much detail comes back: `BOOLEAN_ONLY`, `BASIC`, `SUMMARY`, `COMPLETE`. `COMPLETE` returns *every* unexpected value. On a healthy table nobody notices. On the night a column is wholly malformed, that is the night the results store receives a payload the size of the column, the run slows to a crawl or fails on serialization, and you lose the report about the very failure you needed. Keep the default sampling for scheduled runs and reach for `COMPLETE` deliberately when debugging a specific batch. ## Practical guidance to voice in an interview Say it as a decision procedure: put the validation where the data already is. Table in the warehouse - use the SQL backend so the checks push down and the process stays a thin client. File landing on object storage - use the pandas backend on the file, at file scale. Data already in a Spark job - use the Spark backend so you are not moving it twice. Then scope the batch to the increment, keep an eye on which expectations are the expensive ones, and do not let the result payload become the incident. That answer shows you have thought about the cost of quality checks rather than treating them as free.