In Hadoop MapReduce, what is an InputSplit and what determines its size?

## Logical work, not physical storage An InputSplit is a description of work, produced by the `InputFormat` before any task starts: typically a path, a start offset, a length in bytes, and a list of hosts that hold those bytes. It contains no data. The framework creates exactly one map task per split and passes the split to a `RecordReader`, which is what actually opens the stream and produces key/value pairs. Keeping splits logical is what lets one `InputFormat` sit over HDFS files, object storage, a database table, or a Hive partition without changing the execution model. An HDFS block, by contrast, is physical: a fixed-size chunk of a file stored and replicated on DataNodes. The two coincide by default and are constantly confused in interviews. The split's host list is derived from where the block replicas live, which lets YARN try to place the map container on a node that already holds the data — node-local, then rack-local, then anywhere. ## The size formula `FileInputFormat.computeSplitSize` is `max(minSize, min(maxSize, blockSize))` where: - `mapreduce.input.fileinputformat.split.minsize` — default 1 byte - `mapreduce.input.fileinputformat.split.maxsize` — unset, i.e. effectively unbounded - `dfs.blocksize` — 128 MB by default in Hadoop 2 and 3 With the defaults, the formula collapses to the block size. To get **more, smaller** map tasks, lower `split.maxsize`; to get **fewer, larger** ones, raise `split.minsize` above the block size — which deliberately sacrifices some data locality, since a split then spans blocks that may live on different nodes. Note what you cannot do: `mapreduce.job.maps` is advisory and is ignored by file-based input formats. The number of mappers is a consequence of the split calculation, never a number you set directly. One more detail worth knowing: `FileInputFormat` will not create a tiny trailing split. It applies a slop factor of 1.1, so a remainder no bigger than 10% of a split size is folded into the previous split rather than getting a map task of its own. ## Records that cross a boundary If splits are byte ranges and records are lines, a line will eventually straddle a boundary. `LineRecordReader` handles it with a simple convention: a split whose start offset is not zero skips forward past the first newline (that partial line belongs to the previous split), and every reader keeps reading *past* its own end until it finishes the record in progress. That final read may pull a few bytes from a block on another node — a small, bounded remote read. The consequence is that records are never split or duplicated, but it also means custom `InputFormat`s over record structures without a cheap delimiter must implement this logic themselves or be marked non-splittable. ## Compression changes everything Splittability is a property of the codec, not of the file size: - **gzip** — not splittable. A 10 GB `.gz` file yields one split and therefore one map task that reads all 10 GB, while the rest of the cluster idles. This is one of the most common causes of an inexplicably serial "parallel" job. - **bzip2** — splittable, at a notable CPU cost for compression and decompression. - **LZO** — splittable only if an index file has been generated for it. - **Snappy** — not splittable on its own for raw text, but perfectly fine *inside* a block-compressed container such as SequenceFile, Avro, ORC or Parquet, where the container defines the split points. The practical rule: store large inputs in a container format with internal block compression, or keep individual raw files near the block size. ## The small-files problem, from the job's side Because a split never spans two files in the default `FileInputFormat`, a directory of 50,000 files of 2 MB each produces 50,000 splits and 50,000 map tasks. Each is a container allocation, a JVM start, and a few seconds of real work — the scheduling overhead dwarfs the computation. `CombineFileInputFormat` packs many small files (preferring ones on the same node and rack) into a single split with a target size, which collapses that to a manageable task count. The durable fix is upstream: compact small files before processing. ## What to say in an interview Lead with "logical range, one map task per split, defaults to the block size", then show the formula and immediately name the two things that break the naive picture — non-splittable compression and small files. Those two are what an interviewer is actually probing for, because they are the causes of most surprising map-side parallelism in real clusters.