skip to content

In PHP, how do you read a multi-gigabyte CSV file with fgetcsv() without exhausting memory, and how do you detect the end of the file?

level: middleimportance: must knowfreq 52%

answer

  1. one record per call, not the whole file
  2. open a handle, loop until false
  3. compare with !== false, not feof()
  4. blank line gives [null], not false
  5. length null means unlimited since 8.0

basics

~20 s

Open the file with fopen() and call fgetcsv() in a loop until it returns false; each call parses one record, so memory stays flat. A blank line returns [null], not false, so skip it explicitly.

solid answer

~40 s

Open a handle with `fopen($path, 'r')` and loop `while (($row = fgetcsv($h, null, ',', '"', '')) !== false)`. Each call reads one **record** from the stream and returns it as an indexed array, so only the current row lives in memory, unlike `file()` or `file_get_contents()` plus splitting, which hold the whole file. `fgetcsv()` returns `false` at end of file or on a read error, and a blank line comes back as `[null]`, which you skip. A quoted field containing a line break is read across several physical lines, so record numbers and line numbers can differ. Read the header with the first call and map rows with `array_combine()`, which throws `ValueError` on a count mismatch since PHP 8.0. In PHP 8.4+ pass `escape` explicitly to avoid the deprecation.

code

php · 26 lines
php
<?php
declare(strict_types=1);

$h = fopen('members.csv', 'r');
if ($h === false) {
    throw new RuntimeException('Cannot open members.csv');
}

$header = fgetcsv($h, null, ',', '"', '');
if ($header === false) {
    throw new RuntimeException('members.csv is empty');
}
$n = 0;
while (($row = fgetcsv($h, null, ',', '"', '')) !== false) {
    $n++;
    if ($row === [null]) {
        continue; // blank line
    }
    if (count($row) !== count($header)) {
        error_log("record $n: expected " . count($header) . ' fields');
        continue;
    }
    $member = array_combine($header, $row);
    // insert or validate $member here, then let it go
}
fclose($h);

go deeper

for a junior

Recall the loop shape: fopen, then while fgetcsv returns something other than false. Know that a blank line gives [null].

for a middle

Explain why streaming keeps memory flat, why feof() loops misfire, and how quoted line breaks make records span lines.

for a senior

Show how you keep a long import observable and resilient: peak-memory logging, skipping malformed records instead of aborting, and no accumulation of rows.

for a principal

Weigh where CSV parsing belongs in an import pipeline: a streaming parser in PHP, validation per record, and when a bulk-load path beats row-by-row PHP.

## The problem with loading the whole file A CSV import is often written as `file('members.csv')` or `explode("\n", file_get_contents(...))` followed by a parse of each line. Both build a PHP array holding **every line of the file** before the first row is processed. For a file of a few megabytes that is harmless; for a multi-gigabyte export it hits `memory_limit` and the script dies with a fatal "Allowed memory size exhausted" error. Splitting on `\n` is also wrong for CSV, because a quoted field may itself contain a line break. ## Streaming with fgetcsv() `fgetcsv()` works on an open **stream** (a handle from `fopen()`, `popen()` or `fsockopen()`). Its PHP 8.5 signature is: ```php fgetcsv($stream, ?int $length = null, string $separator = ",", string $enclosure = "\"", string $escape = "\\"): array|false ``` Each call reads **one CSV record** from the current position, parses it into an indexed array of strings, and advances the handle. Memory use is proportional to the longest record, not to the file. Key behaviours: - **`false` means stop.** It is returned at end of file and on a read error, so the loop condition is `($row = fgetcsv(...)) !== false`. - **A blank line is not the end.** It comes back as an array holding a single `null` (`[null]`), and the manual says it is not treated as an error. Skip it with `if ($row === [null]) continue;`. - **Multi-line records.** When a quoted field contains a line break, `fgetcsv()` keeps reading physical lines until the enclosure closes. One call can consume several lines, so "row 1200" in your error messages is a record number, not a line number. - **`length`.** `null` (allowed since PHP 8.0) or `0` means no limit on line length. A positive value must exceed the longest line, or the line is split into chunks; leaving it `null` is the safe default. - **`escape`.** Since PHP 8.4, omitting it raises `E_DEPRECATED` ("the $escape parameter must be provided as its default value will change"). Pass `''` to get plain doubled-quote parsing. ## The feof() trap A common loop is `while (!feof($h)) { $row = fgetcsv($h); ... }`. `feof()` only turns true **after** a read has hit the end, so when the file ends with a newline the last iteration calls `fgetcsv()`, gets `false`, and passes `false` on as if it were a row. Testing the return value of `fgetcsv()` itself avoids the extra iteration. ## Headers and associative rows The first record is usually the header. Read it once before the loop, then build associative rows: 1. `$header = fgetcsv($h, null, ',', '"', '');` 2. inside the loop, `$record = array_combine($header, $row);` 3. validate and hand the record on. Since PHP 8.0, `array_combine()` throws a **`ValueError`** when the two arrays differ in length (it used to warn and return `false`), so a short or long row stops the import unless you check `count($row)` first and log the bad record instead. ## Comparing the approaches | Approach | Memory | Handles quoted line breaks | |---|---|---| | `file()` + `str_getcsv()` per line | whole file | no | | `explode()` on `file_get_contents()` | whole file (twice, briefly) | no | | `fgetcsv()` loop on a handle | one record | yes | | `SplFileObject` with `READ_CSV` | one record | yes | ## Keeping a long import healthy - **Do not accumulate.** Appending every row to an array rebuilds the memory problem; process or write each row, or flush a small batch, then drop it. - **Watch the peak.** `memory_get_peak_usage(true)` logged every few thousand rows shows whether memory is flat. - **Close the handle** with `fclose()` when done, or let it go out of scope. - **Know your delimiter.** A semicolon-separated file read with the default `,` returns each line as one field; pass `separator: ';'` or the right positional argument. ## Any stream works, not only files `fgetcsv()` takes any readable stream, so the same loop serves several sources: - a local file opened with `fopen($path, 'r')`, which returns `false` and emits a warning if the file cannot be opened, so check it before looping; - `php://stdin`, so a CLI import can run as `php import.php < members.csv`; - a `php://temp` stream you filled from a string that arrived in memory. ## A checklist for a production import 1. Open the handle and fail loudly if `fopen()` returned `false`. 2. Read and validate the header; fail if it is `false` (empty file) or missing required columns. 3. Loop on `($row = fgetcsv(...)) !== false`, skipping `[null]` blank lines. 4. Check the field count before `array_combine()` and log bad records with their record number. 5. Process each record and let it go; never collect the file into an array. 6. Close the handle and report counts of read, imported and rejected records.

  • Why is while (!feof($h)) a bug around fgetcsv()?
    `feof()` becomes true only after a read has already hit the end. If the file ends with a newline, the loop runs once more, `fgetcsv()` returns `false`, and the body processes `false` as a row. Putting the `fgetcsv()` call in the condition and comparing with `!== false` stops exactly when there is nothing left.
  • Your import reports errors by line number, but support says the lines do not match. Why?
    `fgetcsv()` counts records, not lines. A quoted field containing a line break makes one record span several physical lines, so the n-th record can start well after line n. Track `ftell()` offsets or count records explicitly, and say "record" in messages.
  • What changes if the header has 5 names and one row has 6 fields?
    Since PHP 8.0, `array_combine()` throws a `ValueError` when the key and value arrays differ in length. Before 8.0 it warned and returned `false`. Check `count($row)` first so one malformed record is logged and skipped instead of aborting the import.

saying these in an interview costs you the question

  • Using file() or file_get_contents() to import a multi-gigabyte CSV
  • Treating a blank line's [null] result as end of file
  • Driving the loop with while (!feof($h)) and not checking fgetcsv's return
  • Believing one fgetcsv() call always reads exactly one physical line
  • Collecting every parsed row into one array before processing