skip to content

Why is ''.join(parts) preferred over building a string with += in a loop?

level: juniorimportance: must knowfreq 70%

answer

  1. Immutable objects cannot grow
  2. Count copies, not appends
  3. One allocation versus n allocations
  4. The in-place resize is not guaranteed
  5. Collect pieces in a list, join once

basics

~20 s

Strings are immutable, so each += builds a brand-new string and copies everything accumulated so far. str.join walks the sequence once in C, sizes the result, allocates it once, and copies each piece exactly once.

solid answer

~50 s

`str` is immutable, so `s += p` cannot extend `s`; it builds a fresh object holding both operands and rebinds the name, which in the general case copies quadratically many characters over n appends. `''.join(parts)` makes one pass to total the lengths, allocates the result once, and copies each piece straight in - and the whole loop runs inside C, so you pay one bytecode instead of n iterations of the interpreter. CPython does carry a fast path that resizes the left operand in place when it is a plain local holding the only reference, which can make a naive `+=` microbenchmark look linear, but that is an unspecified implementation detail: it vanishes the instant anything else holds a reference, and other implementations do not have it. The idiom is to append pieces to a list and join once at the end.

code

python · 10 lines
python
import timeit

setup = "parts = [str(i) for i in range(20000)]"
plus = """
s = ''
for p in parts:
    s += p
"""
print('+=  :', timeit.timeit(plus, setup, number=50))
print('join:', timeit.timeit("''.join(parts)", setup, number=50))

go deeper

for a junior

Recall that str is immutable, so s += p makes a new string every time, and that the idiom is to append pieces to a list and call ''.join(parts) once. Being able to state that plainly is what is being screened for.

for a middle

Explain the mechanics: what += allocates and copies on each pass, what join does in its single C-level pass, and why the interpreter also pays per-iteration overhead the join call does not.

for a senior

Show that you know the CPython in-place resize fast path exists, why a benchmark can therefore mislead, and why code must not depend on it. Mention the bytearray and io.StringIO variants for the shapes where a list plus join is awkward.

for a principal

Own the guidance angle: this is a default-idiom question, not a tuning question. Argue for the shape that has no pathological case rather than for a benchmark result that depends on an unspecified optimisation surviving the next refactor.

### Immutability is the root of it A `str` object in CPython owns a fixed block of storage sized when the object is created. There is no API that appends to an existing `str`, and no dunder that mutates one: `str` implements `__add__` but not a mutating `__iadd__` of its own. So `s += p` compiles to an in-place-add opcode that, finding no mutating implementation, falls back to `s = s + p`. That expression allocates a new string of `len(s) + len(p)` characters and copies both operands into it. The old value becomes garbage. Run that n times and the copying adds up: the first append copies 1 unit, the second 2, the third 3, and the total is on the order of n squared characters moved for n appends. That is the classic reason the pattern collapses on large inputs while looking perfectly innocent on ten items. ### What join does instead `''.join(parts)` is a single call into C. It materialises the argument into a sequence if it is not one already, walks it once to compute the total length and the widest character representation required, allocates exactly one result object, and then copies each piece into place with a block copy. Every element is copied once. There are no intermediate strings, no garbage, and - just as important for this leaf - no per-element interpreter work: the iteration, the length arithmetic and the copying all happen below the bytecode level. The `+=` loop pays a `FOR_ITER`, a name load, an in-place-add dispatch and a store on every single iteration; `join` pays one call. ### The CPython fast path, and why you must not rely on it If you benchmark the two, the `+=` loop often looks far better than quadratic. CPython's evaluation loop special-cases in-place addition of two `str` objects: it clears its own stack reference to the left operand first, and when the reference count then drops to one - meaning that local name is the only thing pointing at the string - it reallocates the existing buffer in place and appends. No copy of the accumulated prefix happens, and the loop behaves close to linear. This is an optimisation, not a language guarantee. It disappears the moment a second reference exists: store the running value in a list, close over it, hand it to a function that keeps it, assign it to an attribute or a dict entry, or simply keep the previous value in another variable, and the refcount is above one and every append copies again. It also does not apply where the accumulator is not a simple local, and other Python implementations - which may not use reference counting at all - have nothing like it. Code whose performance silently depends on an unspecified fast path is code that will fall off a cliff during a refactor that looks harmless in review. ### The shapes to reach for - Collect the pieces in a `list` and call `''.join(parts)` once. `list.append` is amortised constant time on a growable array of pointers, and the join at the end is one allocation. - If the pieces are produced by scattered write-style calls rather than a single loop, `io.StringIO` gives a file-like accumulator and `getvalue()` performs the equivalent join. - For binary data the mutable type does the job directly: `bytearray` supports genuine in-place extension, so `buf += chunk` on a `bytearray` really is amortised constant time, and `b''.join(chunks)` is the equivalent one-shot form for `bytes`. - `str.join` requires every item to be a `str`. Joining a list of numbers raises `TypeError`; map them to `str` first, or use a comprehension that formats them. ### Measuring it honestly A microbenchmark must defeat the fast path if you want to see the real cost model, otherwise you measure the optimisation rather than the semantics. Keeping a second reference to the accumulator inside the loop is enough. And the usual caveat applies: this only matters when string building is actually hot. In a function that also parses input or touches the network, the concatenation may be noise - profile before rewriting. What is not negotiable is the idiom itself: `join` is shorter, clearer and never has the pathological case, so there is no situation where the `+=` loop is the better default.

  • Why does a += loop over strings sometimes benchmark as if it were linear on CPython?
    CPython's evaluation loop special-cases in-place addition of two `str` objects. It drops its own stack reference to the left operand, and if the reference count then falls to one the buffer is reallocated and extended in place instead of a new object being built. That makes the loop close to linear - until any second reference to the accumulator exists, at which point every iteration copies the whole prefix again. It is an unspecified implementation detail, not a guarantee.
  • What is the equivalent idiom when you are accumulating bytes rather than str?
    Use a `bytearray`, which is genuinely mutable: `buf += chunk` extends the existing buffer in amortised constant time, with no dependence on reference counts. Call `bytes(buf)` at the end if an immutable result is needed. If you already have all the pieces, `b''.join(chunks)` is the one-shot equivalent of `str.join` and does a single allocation.
  • When would you reach for io.StringIO instead of a list plus join?
    When the pieces are produced by scattered write-style calls rather than one tight loop - a serializer or a formatter that passes a sink around. `io.StringIO` gives a file-like object with `write`, so the same code can target a real file later, and `getvalue()` does the join at the end. The cost is comparable to list-plus-join; the win is the interface.

Repeated += is photocopying the whole manuscript each time you add a page; join counts the pages first, prints the book once, and drops each page into place.

saying these in an interview costs you the question

  • Claims str objects are mutable in CPython
  • Says += in a loop is always fine because CPython optimizes it
  • Thinks join is faster only because it is shorter code
  • Cannot say why repeated concatenation copies more than once
  • Believes str.join accepts a list of arbitrary objects

context

resolving links…