skip to content

What does Python's open() do to line endings when newline is left at its default?

level: juniorimportance: should knowfreq 38%

answer

  1. Text mode does more than decode
  2. Three conventions, one in-memory form
  3. Read normalizes, write platform-izes
  4. The default is newline=None
  5. os.linesep appears on the write side

basics

~10 s

With the default newline=None, open() reads in universal-newline mode: '\r\n', '\r' and lone '\n' all arrive in your string as '\n'. On write, every '\n' you emit is translated to os.linesep.

solid answer

~40 s

A file opened in text mode is wrapped in an `io.TextIOWrapper`, which decodes bytes to `str` **and** translates line endings. With the default `newline=None` this is universal-newline mode: on read, any of `'\r\n'`, `'\r'` or `'\n'` in the file becomes a single `'\n'` in the string you get back, so your code never has to know which convention wrote the file. On write it runs the other way: every `'\n'` you pass to `write()` is replaced with `os.linesep`, which is `'\r\n'` on Windows and `'\n'` elsewhere. Pass `newline=''` to keep the original characters, or `newline='\r\n'` to force CRLF output on every platform. Binary mode (`'rb'`/`'wb'`) does none of this — it moves bytes through untouched, and passing `newline=` with a binary mode raises `ValueError`.

code

python · 12 lines
python
import os
import tempfile

path = os.path.join(tempfile.mkdtemp(), "data.txt")
with open(path, "wb") as f:
    f.write(b"a\r\nb\rc\n")

with open(path, encoding="utf-8") as f:            # newline=None (the default)
    print(repr(f.read()))                          # 'a\nb\nc\n'

with open(path, encoding="utf-8", newline="") as f:  # translation switched off
    print(repr(f.read()))                          # 'a\r\nb\rc\n'

go deeper

for a junior

Be ready to say what the default does in both directions: reading normalizes every convention to '\n', writing turns '\n' into the platform ending. Knowing that binary mode skips all of it is enough at this level.

for a middle

Explain the mechanics: a TextIOWrapper decodes and translates, and name the accepted newline values and what each does on read versus write. Expect to be asked which one preserves the original characters.

for a senior

Show that you know when the translation is a hazard — byte-exact round-trips, checksums, formats with their own terminators — and that you reach for read_bytes() to diagnose rather than guessing from the decoded string.

for a principal

Own the convention: pick one policy for the codebase (explicit encoding, explicit newline at every text boundary, binary for anything byte-exact) so platform differences never become a class of bug the team rediscovers per file format.

## Two jobs hiding behind one call `open(path)` in text mode returns an `io.TextIOWrapper` sitting on top of a raw byte stream, and that wrapper does two separate transformations. The first is the codec: bytes are decoded to `str` on the way in and encoded on the way out. The second, much less visible, is **newline translation**. Both are on by default, and both can change what you observe relative to what is on disk. This section is about the second one. Three conventions for ending a line are still in circulation. Unix-descended systems use a single line feed, `'\n'` (0x0A). Windows, DOS and a long list of network protocols use a carriage return followed by a line feed, `'\r\n'` (0x0D 0x0A). Classic Mac OS used a bare carriage return, `'\r'`. Python's answer is not to pick a winner but to normalize: inside your program a line ends with `'\n'` and nothing else, and the boundary layer converts. ## Reading With `newline=None` — the default — the wrapper is in universal-newline mode. It recognises all three sequences as line terminators, and it **replaces every one of them with a single `'\n'`** before your code sees the characters. That is true for `read()` as well as for iteration and `readlines()`, so a file written on Windows and a file written on Linux produce identical strings. It also means the string you get back is not a faithful picture of the file: a 13-byte file can decode to a 12-character string with no error and no warning. The `newline` argument on read accepts exactly five values, and they split into two behaviours: * `None` — recognise all three, translate them all to `'\n'`. * `''` — still recognise all three as line boundaries (so iterating the file splits in the right places), but **translate nothing**; the original characters stay in the string. * `'\n'`, `'\r'` or `'\r\n'` — only that exact string terminates a line, and again nothing is translated. Any other value raises `ValueError`. The distinction between `None` and `''` is the one that matters in practice: both let you iterate lines sensibly, but only `''` preserves the bytes' shape. ## Writing The write side mirrors it. With `newline=None`, every `'\n'` in the string you pass to `write()` is replaced by `os.linesep` — `'\r\n'` on Windows, `'\n'` on Linux and macOS. This is why the same script emits CRLF files on one machine and LF files on another without a line of platform code. With `newline=''` or `newline='\n'` no translation happens at all and your `'\n'` reaches the disk as one byte. With `newline='\r'` or `newline='\r\n'` each `'\n'` becomes that sequence, on every platform — which is how you deliberately produce CRLF output from Linux. Note what the write side does *not* do: it only ever looks at `'\n'`. A `'\r'` you put in the string yourself is passed through untouched, which is exactly how files end up with `'\r\r\n'` — you supplied the `'\r'`, and text mode added its own after the `'\n'`. ## Binary mode opts out entirely `open(path, 'rb')` and `open(path, 'wb')` return byte streams with no wrapper, no codec and no newline translation; the bytes on disk are the bytes you get. Because there is nothing to translate, combining a binary mode with the argument is rejected outright: `open(path, 'rb', newline='')` raises `ValueError: binary mode doesn't take a newline argument`. When the exact bytes matter — hashing, signing, copying, uploading, or any format that is not line-oriented text — binary mode is the correct tool, not a cleverer `newline=` value. ## A neighbouring trap: splitlines Universal-newline mode recognises three sequences. `str.splitlines()` recognises far more: besides `'\n'`, `'\r'` and `'\r\n'` it also splits on vertical tab, form feed, the file/group/record separators, and the Unicode characters NEL, LINE SEPARATOR and PARAGRAPH SEPARATOR. So `text.splitlines()` can produce more lines than iterating the same file did. When you need exactly the file's notion of a line, iterate the file object or use `str.split('\n')`; reach for `splitlines()` when you want the generous Unicode definition. ## What to actually do For ordinary text — reading a config file, writing a report — leave `newline` alone and enjoy the normalization. Reach for `newline=''` when another layer already emits its own terminators or when embedded carriage returns are data. Reach for a binary mode when the file is not text at all. And when a file surprises you, look at `pathlib.Path(p).read_bytes()` rather than `read_text()`: the translation is invisible in the string and obvious in the bytes.

  • How do you read a file so the original '\r\n' survives into the string?
    Pass `newline=''` to `open()`. Line boundaries are still recognised, so iterating the file yields sensible lines, but no characters are replaced and each line keeps its `'\r\n'` ending. `newline='\n'` also avoids translation but changes what counts as a terminator, so a bare `'\r'` no longer ends a line. Binary mode gives you the bytes with no decoding at all.
  • What happens if you pass newline='' together with a binary mode?
    `ValueError: binary mode doesn't take a newline argument`. Binary streams neither decode nor translate, so the argument is meaningless there; the same applies to `encoding=` and `errors=`. If you find yourself wanting it, you either want text mode with `newline=''` or you want plain bytes.
  • Does str.splitlines() split on the same set of endings as universal-newline reading?
    No — it splits on a wider set. Universal-newline mode recognises `'\n'`, `'\r'` and `'\r\n'`; `str.splitlines()` additionally splits on vertical tab, form feed, the file/group/record separators and the Unicode NEL, LINE SEPARATOR and PARAGRAPH SEPARATOR characters. A string containing one of those yields more lines than the file's own iteration produced.

Text mode is a customs desk: whatever line-ending currency the file arrives in, you are handed the local one, and on the way out it is converted back to whatever the destination platform spends.

  • Says Python only translates line endings on Windows
  • Believes text-mode reading preserves the file's exact bytes
  • Thinks newline=None means no translation at all
  • Assumes newline='' turns off line splitting as well
  • Passes newline='' alongside a binary mode like 'rb'
  • Treats str.splitlines() as identical to iterating the file

Universal-newline text mode has behaved this way since Python 3.0 and is unchanged through 3.14. Related conveniences arrived later: pathlib.Path.write_text() gained a newline parameter in 3.10 and Path.read_text() gained one in 3.13.

context