How does Redshift's COPY spread work across slices, and how should you lay out the S3 files?

## The unit of parallelism is the file When you run a `COPY`, the leader node resolves the source into a list of S3 objects — either by listing everything under the prefix you gave, or by reading the file list out of a manifest. It then distributes that list across the cluster's **slices**. Each slice opens its own connections to S3 and loads its own files. Data never flows through the leader node. A slice is the unit of parallel work inside Redshift: each compute node is divided into slices, each with a share of CPU, memory and local storage, and each owning a portion of every table's rows. How many slices a node has depends on the node type, and the total is nodes multiplied by slices per node; `STV_SLICES` shows the mapping for a running cluster. Because work is handed out per file, **the file layout in S3 determines how much of the cluster participates in the load**. ## Why one big compressed file is the worst case Gzip, bzip2, lzo and zstd streams cannot be decompressed from an arbitrary offset, so a compressed file is an indivisible unit of work. Point `COPY` at one 50 GB gzip file and one slice does all the reading, decoding and writing while every other slice waits. Doubling the cluster size changes nothing. This is the single most common reason a load is far slower than the cluster should manage. The practical rule: never rely on a single file being divided for you. Split the data yourself. ## Sizing the files AWS's published guidance for delimited-text loads is: - Files of **roughly equal size** — a skewed set means the slice holding the biggest file finishes last, and the whole `COPY` takes as long as its slowest slice. - **About 1 MB to 1 GB per file after compression.** Below that, per-object overhead dominates; above it, you lose granularity for balancing. - **A file count that is a multiple of the number of slices**, so every slice gets the same number of rounds. With 16 slices, 32 files means two rounds each; 17 files means one slice does two rounds while 15 sit idle for the second. The small-file failure mode matters as much as the big-file one. A prefix holding 200,000 files of 20 KB each spends most of the load doing S3 requests and open/close work rather than moving data. ## Manifests: telling COPY exactly what to load Instead of a prefix, you can hand `COPY` a manifest — a JSON document listing the exact objects to load — with the `MANIFEST` option: ```json { "entries": [ {"url": "s3://landing/2026-01-01/part-00.parquet", "mandatory": true}, {"url": "s3://landing/2026-01-01/part-01.parquet", "mandatory": true} ] } ``` ```sql COPY events FROM 's3://landing/manifests/2026-01-01.json' IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftLoad' FORMAT AS PARQUET MANIFEST; ``` This buys three things. First, **exactness**: the load consumes the files your pipeline says it produced, not whatever happens to be sitting under a prefix, so a half-written batch or a stray file from another job cannot leak in. Second, **files from several prefixes or buckets** can be loaded as one unit. Third, `mandatory` makes a missing object fail the load loudly instead of silently loading less data than expected. ## Distribution during the load Parallel reading is only half the story. Once a slice has parsed rows, they must go to the slice that owns them under the table's distribution style, which means rows may be sent across the network mid-load. That is normal and usually cheap relative to the I/O, but it does mean the load's cost is not purely a function of file layout — a badly skewed distribution key concentrates the *writing* on a few slices even when the *reading* was perfectly balanced. ## Failure handling A `COPY` is one transaction: if it fails, nothing is left half-loaded. Rejected rows are recorded with the offending file, line and column, which you read from `STL_LOAD_ERRORS`. The `MAXERROR` option lets a load tolerate a bounded number of bad rows rather than aborting — useful for messy third-party feeds, dangerous as a default, because it turns a data-quality alarm into a silent partial load. ## Putting it together A healthy load looks like: a producer writing evenly sized, compressed part files into a dated prefix; a manifest naming exactly that batch; one `COPY` per batch; and monitoring on rejected rows. The failure cases to recognise instantly are one giant compressed file, a prefix full of kilobyte-sized fragments, and a file count that leaves most slices idle.