Why is str.split(',') wrong for parsing a CSV line, and what does csv.reader do instead?
answer
- Commas are not always separators
- Quoting is what breaks naive parsing
- One record is not one line
- Doubled quote characters, not backslashes
- csv.reader yields lists of str
basics
~20 sA comma inside a quoted field is data, not a separator, so str.split(',') tears such a row apart. csv.reader runs the quoting state machine over the file object and yields each record as a list of strings.
solid answer
~50 s`str.split(',')` treats every comma as a separator, but a CSV field may legally contain a comma, a doubled quote character or even a line break as long as the field is quoted. Splitting therefore silently produces the wrong number of columns on exactly the rows that matter, and the damage shows up far downstream. `csv.reader(fileobj)` wraps the file object in a parser that tracks whether it is inside a quoted field, un-doubles embedded quote characters, and pulls extra lines from the underlying iterator when a quoted field spans them. It yields one `list[str]` per record. Every value comes back as `str` - the module does no type conversion unless you ask for `csv.QUOTE_NONNUMERIC`, which converts unquoted fields to `float` on read. Delimiter and quote character are configurable via `delimiter=` and `quotechar=`, so the same parser reads tab- or semicolon-separated files.
code
python · 6 linesimport csv
import io
raw = 'city,note\n"Paris, FR","said ""oui"" here"\n"Berlin, DE","two\nlines"\n'
for row in csv.reader(io.StringIO(raw, newline="")):
print(row)go deeper
Be ready to say why a comma inside a quoted field defeats str.split(',') and to reach for csv.reader by reflex. Remember that every field comes back as a string, so you do your own int or float conversion.
Explain the mechanics: the reader is a lazy iterator over the file object, it un-doubles embedded quote characters, and it pulls extra lines when a quoted field spans them. Know delimiter, quotechar and the QUOTE_ constants by name.
Show judgement about data you did not produce: quoting anomalies, ragged rows and files whose dialect differs from the one documented. Say when you would register a named dialect rather than repeat keyword arguments across a codebase.
Own the boundary question: the csv module gives you correct record shape and nothing else. Decide where schema validation, typing and rejection of bad rows live, and whether a self-describing format would serve the pipeline better than CSV at all.
### The failure mode CSV looks like a format you can parse with one line of string manipulation, and for hand-made sample data you can. The trouble is that the files you actually receive are produced by spreadsheets, database exports and other people's programs, all of which quote a field the moment it contains something structural. A row such as `"Paris, FR",7` has two fields; `str.split(',')` on that text returns three (`'"Paris'`, `' FR"'`, `'7'`), with a stray quote character welded onto two of them. Nothing raises. The row simply becomes wrong, and it stays wrong until some later stage trips over a column count or a value that will not convert. Three things a quoted field can hold defeat naive splitting: - **The delimiter itself.** `"Paris, FR"` is one field containing a comma. - **The quote character.** It appears doubled inside the field: `"said ""oui"" here"` is the single value `said "oui" here`. Note that the convention is doubling, not backslash escaping - a candidate who reaches for `\"` is thinking of a different format. - **A line break.** A quoted field may contain a newline, which means one *record* is not the same thing as one *line* of the file. Any approach built on iterating lines and splitting them is structurally unable to handle this. ### What csv.reader actually is `csv.reader(f)` takes any iterator of strings - usually an open text file, sometimes an `io.StringIO`, sometimes a generator - and returns a reader object that is itself an iterator. Each `next()` consumes as many strings from the source as the record needs and returns a `list` of the record's fields. The parsing is done by a C state machine, so it is both correct and considerably faster than a Python-level loop. Because the reader is a lazy iterator over the source, it never holds more than the current record in memory, and it is only valid while the underlying file object is open. Returning a reader out of a `with open(...)` block is a classic bug: the file closes and the next iteration raises `ValueError`. ### Types, and the absence of conversion Every field arrives as `str`, including `'7'` and `''`. An empty field is the empty string, not `None`. This is deliberate: the module refuses to guess. If you want numbers you convert them yourself, which is also where you get to decide what a malformed value should do. The one built-in exception is `csv.QUOTE_NONNUMERIC`, which on *reading* converts every unquoted field to `float` and raises `ValueError` if that fails. ### The writing side `csv.writer(f).writerow(seq)` is the mirror image: it applies quoting so that the row can be read back correctly. The `quoting=` argument takes the module's constants and decides how aggressive that is. `csv.QUOTE_MINIMAL`, the default, quotes only fields that need it - those containing the delimiter, the quote character or a newline. `csv.QUOTE_ALL` quotes everything, which is useful when a downstream consumer is itself sloppy. `csv.QUOTE_NONE` quotes nothing and raises if a field would need quoting, unless you supply an `escapechar=`. `csv.QUOTE_NONNUMERIC` quotes every non-numeric field. Python 3.12 added two more, `csv.QUOTE_STRINGS` and `csv.QUOTE_NOTNULL`, which differ in how they treat `None` versus the empty string on output. ### Dialects The bundle of delimiter, quote character, quoting policy, line terminator and whitespace handling is a *dialect*. `csv.excel` is the default; `csv.excel_tab` and `csv.unix_dialect` ship alongside it, and `csv.register_dialect` names your own so that reader and writer cannot drift apart. Pass either the dialect name as the second argument or the individual keywords. `csv.Sniffer` will guess a dialect from a text sample - handy for exploratory work, unreliable enough that you should not put it in an unattended pipeline. ### The honest limits The module parses; it does not validate. Ragged rows come back as short or long lists without complaint (unless you use `csv.DictReader`, which has `restkey=` and `restval=` for exactly this). It has no schema, no header semantics beyond what you impose, and no notion of types. What it buys you is that the shape of every record is correct, which is precisely the thing hand-rolled splitting cannot promise. ### Where the reader gets its input One detail pays off later: `csv.reader` does not require a file. It accepts any iterator of strings, which means a list of lines, an `io.StringIO`, a generator that decodes a network stream, or a generator that filters comment lines out before the parser sees them all work identically. That is the idiomatic way to skip a preamble - wrap the file in a generator expression that drops the junk lines and hand the generator to `csv.reader` - and it is far safer than trying to strip the preamble with string surgery afterwards, because the parser still owns the record structure. The one hard constraint is that those strings must be `str`. Opening the file in binary mode and passing it to `csv.reader` fails, because the parser has no encoding to apply. Decoding is a separate layer's job, and keeping it separate is what lets the same reader code serve a file, a socket and a test fixture without change.
- What Python types do the fields of a csv.reader row have?All of them are `str`, including digits and empty fields - an empty field is `''`, never `None`. The module does no conversion, so you cast columns yourself and decide what a bad value means. The single exception is `csv.QUOTE_NONNUMERIC`, which on reading converts unquoted fields to `float` and raises `ValueError` when one will not parse.
- What does csv.QUOTE_MINIMAL do differently from csv.QUOTE_ALL when writing?`csv.QUOTE_MINIMAL` is the writer default and quotes a field only when it has to - when the field contains the delimiter, the quote character, or a line break. `csv.QUOTE_ALL` wraps every field regardless. Neither changes what a correct reader gets back; QUOTE_ALL just produces larger files that survive sloppier downstream consumers.
- How do you read a semicolon-separated or tab-separated file with the same code?Pass `delimiter=';'` or `delimiter='\t'` to `csv.reader`, or name a dialect. The module's parser is not comma-specific - delimiter, `quotechar=`, quoting policy and line terminator are all dialect settings. Registering a named dialect with `csv.register_dialect` is better than repeating keywords, because it keeps the reader and writer of the same file in step.
Splitting on commas is like cutting a sentence at every space and expecting to get words back - it works until someone writes a phrase in quotation marks and you slice straight through it.
saying these in an interview costs you the question
- Claims split(',') is fine when the data is clean
- Thinks csv.reader converts numeric columns to int or float
- Believes a CSV field can never contain a newline
- Escapes an embedded quote with a backslash instead of doubling it
- Writes a regular expression to parse quoted fields
- Assumes an empty field comes back as None