Great Expectations

## What Great Expectations is Great Expectations (GX) is an open-source Python framework for expressing data-quality rules as **declarative assertions** and running them as a gate inside a data pipeline. Rather than scattering ad-hoc checks like `assert df.order_id.notnull().all()` through transformation code, you declare what the data must look like, store that declaration as a reviewable artifact, and run it against data at a pipeline step or on a schedule. ## An Expectation An Expectation is a single named assertion plus keyword arguments - for example `expect_column_values_to_be_between(column="amount", min_value=0)`. They come in families: - **Column map expectations** evaluate row by row: `expect_column_values_to_not_be_null`, `expect_column_values_to_be_unique`, `expect_column_values_to_be_between`, `expect_column_values_to_match_regex`, `expect_column_values_to_be_in_set`. - **Column aggregate expectations** evaluate one statistic over the column: `expect_column_mean_to_be_between`, `expect_column_distinct_values_to_be_in_set`. - **Table expectations** describe the whole batch: `expect_table_row_count_to_be_between`, `expect_table_columns_to_match_ordered_list`. Every expectation is serializable - the expectation name plus its kwargs - which is precisely why a suite reads as a spec rather than as code. ## The Expectation Suite A suite is a named collection of expectations scoped to one dataset or asset (say, `orders_raw`). It lives in the **Data Context** - GX's entry point and configuration root, either backed by a project directory on disk or created ephemerally in memory - so suites can be stored beside the pipeline, diffed, and reviewed. You author a suite three ways: by hand, interactively against a sample batch while you inspect it, or by profiling a sample so GX proposes expectations from observed ranges and distributions. A profiled suite is a **draft, not a contract**: unedited profiler output encodes accidents of whatever sample it saw, and the usual failure is a suite full of over-tight ranges that fires every Monday. ## What validation produces Validation applies a suite to a **batch** - a defined slice of data, such as one day's partition of a table or one landed file. The output is a validation result for each expectation and an overall suite-level result: - top-level `success` - false if any expectation failed; - per expectation: its own `success`, the expectation's configuration, and a `result` dictionary with `element_count` (rows evaluated), `unexpected_count`, `unexpected_percent`, and a bounded `partial_unexpected_list` of actual offending values. The verbosity of that block is controlled by `result_format`, which ranges from `BOOLEAN_ONLY` through `BASIC` and `SUMMARY` to `COMPLETE`. `COMPLETE` returns every unexpected value, which is wonderful for debugging and dangerous on a wide failure: a column that is entirely wrong will serialize millions of values into the result store. ## Data Docs GX renders suites and validation results into **Data Docs**, a static HTML site: what we expect of this dataset, and what the last runs found. It is the artifact you point an analyst at when they ask why the load was blocked. It is documentation and evidence, not a monitoring system - it has no alerting of its own. ## What a suite does not do by itself Three things trip newcomers up. First, running a suite does not stop anything - somebody must read `success` and act on it. Second, GX validates; it never repairs data. Third, GX does not schedule itself - an orchestrator task, a job step, or a CI job invokes it. ## In an interview Be able to say the vocabulary chain out loud: Data Context holds the configuration; a data source and asset define where the data is; a batch is the slice being checked; an Expectation Suite is the set of rules; validation produces a Validation Result; a Checkpoint is the runnable unit that ties suite to batch and fires actions; Data Docs is the rendered report. Being able to place *where validation executes* in a pipeline is the point of learning the tool's nouns at all.