skip to content

What does schema inference cost when Spark reads a large CSV or JSON directory?

level: middleimportance: should knowfreq 45%

answer

  1. text files carry values, not types
  2. something runs before your first action
  3. today's data decides tomorrow's types
  4. the footer already knows, for some formats

basics

~20 s

It costs an extra full pass over the data before your query even starts, and it makes the schema depend on today's values, so a column can silently change type between runs. Supplying an explicit StructType removes both problems.

solid answer

~50 s

Text formats carry no schema, so Spark has to look at the data to build one. `spark.read.json(path)` scans the files and launches a job at *definition* time, before you call any action; for CSV, `inferSchema` defaults to false — every column comes back as a string — and turning it on adds a separate pass over the input. So a read that looks lazy actually reads the data twice. Worse, the schema becomes data-dependent: an id column that is all digits today infers as a numeric type, and one non-numeric value tomorrow makes it a string, breaking everything downstream. The fix is to declare the schema explicitly with `.schema(...)`, which skips the inference job and pins the contract. Self-describing formats such as Parquet and ORC avoid the problem entirely because the schema is in the file footer — though enabling `mergeSchema` makes Spark read every file's footer.

code

python · 13 lines
python
from pyspark.sql.types import StructType, StructField, StringType, DecimalType, TimestampType

schema = StructType([
    StructField("order_id", StringType(), nullable=False),
    StructField("amount",   DecimalType(18, 2), nullable=True),
    StructField("ts",       TimestampType(), nullable=True),
])

df = (spark.read
        .schema(schema)               # no inference job at all
        .option("header", "true")
        .option("mode", "FAILFAST")
        .csv("s3://bucket/orders/"))

go deeper

for a junior

Know that CSV and JSON have no built-in schema, that CSV columns arrive as strings unless you ask otherwise, and that you can pass a schema yourself.

for a middle

Explain the mechanics: JSON infers by default and CSV's inferSchema defaults to false, either way inference costs an extra pass and fires a job before any action.

for a senior

Argue the correctness case, not just the cost — pinned types, explicit DECIMAL and TIMESTAMP choices, a deliberate parse mode, and a quarantine path for malformed rows in a production ingest.

for a principal

Treat the schema as a contract between producers and consumers: where it is declared, how changes are versioned and rolled out, and whether a table format with catalog-managed schema removes the class of failure entirely.

## Why inference exists CSV and JSON files carry values, not types. `"1042"` in a CSV could be a string, an integer or a zero-padded account code; `{"qty": 3}` in one JSON record and `{"qty": "3"}` in another are legitimately different. To hand you a typed DataFrame, Spark must either be told the schema or go and look. ## The two behaviours, precisely **JSON.** `spark.read.json(path)` infers by default. It reads the files, unions the fields it finds, and widens conflicting types where it can. Because this happens while you build the DataFrame, it launches a Spark job *before* your first action — one of the few places Spark's laziness does not hold. The `samplingRatio` option (default 1.0, meaning all records) lets you infer from a fraction, trading accuracy for time. **CSV.** `inferSchema` defaults to **false**, so `spark.read.csv(path)` returns every column as `StringType` unless you say otherwise. Setting `option("inferSchema", "true")` makes Spark read the input an extra time to determine types. Note `header` also defaults to false, so without it your first data row is column `_c0`, `_c1`, and so on. Either way, on a directory of thousands of files in object storage, that is a full extra scan of the data before any of your logic runs — often the single largest avoidable cost in an ingestion job. ## The correctness problem is worse than the cost A data-dependent schema is a contract that changes without warning. Concretely: - An id column of digits infers as `LongType` today; one value containing a letter arrives tomorrow and the column becomes `StringType`, so a downstream join on it fails or, worse, matches nothing. - Leading zeros are destroyed the moment a code column is inferred as numeric. - A price column inferred as `DoubleType` introduces floating-point behaviour where you wanted `DecimalType(18,2)`. - Inference cannot express business nullability: a column that must never be null is inferred nullable simply because Spark saw no nulls in the sample — or the reverse. - With a partitioned directory, different files can produce different inferred types, and the merged result may not be what either file meant. An explicit schema turns all of these into loud, immediate failures instead of silent drift. ## Declaring the schema Build a `StructType` (or pass a DDL string such as `"order_id STRING, amount DECIMAL(18,2), ts TIMESTAMP"`) and hand it to `.schema(...)` before the format call. Inference is skipped entirely — no extra job — and the read enforces your intent. Pair it with `mode`: `PERMISSIVE` is the default and puts unparseable rows' raw text in `_corrupt_record` (or nulls the fields), `DROPMALFORMED` discards them, `FAILFAST` aborts the read. For a production ingest, an explicit schema plus a deliberate mode choice is the baseline, and capturing `_corrupt_record` gives you a quarantine path rather than silent data loss. ## Self-describing formats change the picture Parquet and ORC store the schema in each file's footer, so reading is cheap and typed without any inference pass. Two caveats worth naming: - **Schema merging.** `spark.sql.parquet.mergeSchema` is off by default. Enabling it (or the equivalent read option) makes Spark read the footer of every file to build a union schema, which on a large directory is itself a costly metadata job. - **Partition column inference.** Directory-style partitioning (`/dt=2026-08-21/`) still has its types inferred from the path values, which is why a partition column of numeric-looking strings can arrive as an integer. Table formats with a catalog — Delta, Iceberg, Hudi, or a Hive metastore table — remove even the footer scan, because the schema is metadata the engine reads once. ## What a strong answer sounds like "Text formats have no schema, so Spark either infers by scanning — an extra pass that fires a job before any action, and for JSON happens by default — or takes the schema I give it. In production I always pass an explicit StructType with a deliberate DECIMAL and TIMESTAMP choice and a FAILFAST or quarantine mode, both to save the pass and because an inferred schema is a contract that silently changes with the data. For Parquet the schema is in the footer, so the only thing to watch is mergeSchema." ## The operational tell If you see a Spark job in the UI whose only description is a read, running before the job you expected, that is inference. It is one of the quickest wins available in a slow ingestion pipeline.

  • How would you notice in the Spark UI that inference is costing you?
    You see a job running before the one your action triggered, described only as a read of the input path, and its input bytes roughly match the dataset size. That extra scan is the inference pass, and supplying an explicit schema makes it disappear entirely.
  • What does mode PERMISSIVE do to a row that does not match the schema you supplied?
    PERMISSIVE is the default: it does not fail, it nulls the fields it cannot parse and, if you declare a `_corrupt_record` column in the schema, puts the raw text there. That makes a quarantine path possible. `DROPMALFORMED` silently discards such rows and `FAILFAST` aborts the read.
  • Parquet stores its schema in the footer, so is schema handling free there?
    Almost. A single read uses the footer and costs nothing extra. But `spark.sql.parquet.mergeSchema`, which is off by default, makes Spark read every file's footer to build a union schema — expensive on a large directory. Directory partition column types are also still inferred from the path values.

Inferring a schema is guessing a form's field types by reading every submission that arrived today. It works until tomorrow's post brings a letter in the phone-number box.

saying these in an interview costs you the question

  • Says spark.read.csv infers types by default
  • Thinks schema inference is lazy like other transformations
  • Believes an inferred schema is stable across runs
  • Claims an explicit schema only helps performance, not correctness
  • Assumes Parquet reads always merge schemas across files

context

resolving links…