Why does a Python list of 10 million float readings use several times the raw data size?
answer
- Two costs, not one
- Slots point at objects elsewhere
- Every reading is its own heap object
- Roughly 24 bytes per float object
- getsizeof on the list misses the bigger term
basics
~20 sA list stores eight-byte pointers rather than the numbers, so ten million elements is about 80 MB of slots plus over-allocated slack. Each reading is also a separate float object of about 24 bytes, adding roughly 240 MB more.
solid answer
~50 sTwo costs, not one. The list itself is a contiguous array of object *pointers*: eight bytes per slot on a 64-bit build, so ten million elements is 80 MB, plus roughly an eighth of slack if it was grown by `append`. That is the only part `sys.getsizeof` on the list reports. The readings are separate heap objects — a float is a refcount, a type pointer and the eight-byte double, about 24 bytes each — so ten million of them adds around 240 MB that the list merely points at. Total near 320 MB against 80 MB of actual doubles, with nothing leaking. Identical readings are not shared either: small integers are cached, floats are not. In a sensor-telemetry collector the fix is not to hold ten million readings: fold into running aggregates, bound the window, or store values in a compact fixed-width buffer.
code
python · 10 linesimport sys
readings = [i * 0.5 for i in range(1_000_000)]
slots = sys.getsizeof(readings)
payload = sum(sys.getsizeof(value) for value in readings)
print(f"list object: {slots / 1e6:8.2f} MB")
print(f"float objects: {payload / 1e6:8.2f} MB")
print(f"total: {(slots + payload) / 1e6:8.2f} MB")go deeper
Recall that a Python list holds references to objects rather than raw numbers, so a list of numbers costs the pointers plus a full object for each value, not just the bytes of the values.
Be able to do the arithmetic out loud: eight bytes per slot plus over-allocation, and roughly 24 bytes for each float object, and explain why sys.getsizeof on the list only shows the first term.
Show diagnosis and judgement together: confirm with allocation tracing rather than guessing, reject the cosmetic fix of rounding values, and reach first for not materializing the data — aggregate or bound the window before optimizing storage.
Own the data-path decision: whether high-volume readings should ever reach a Python container as individual objects, what the retention window is worth, and where the boundary sits between in-process buffering and downstream storage.
### Do the arithmetic before you debug A collector buffering ten million float readings in a Python list is often reported as a memory leak. It usually is not. It is the expected footprint, and the expectation was wrong. The arithmetic has exactly two terms. **Term one — the list's pointer array.** A CPython list stores object *pointers*, eight bytes each on a 64-bit build, in one contiguous array. Ten million elements is 80 MB of slots. If the list was grown by repeated `append` there is also over-allocated slack of roughly an eighth, so call it 80–90 MB. `sys.getsizeof(readings)` reports this term and nothing else. **Term two — the float objects themselves.** Every reading is a separate heap object: reference count, type pointer, and the eight-byte double, which is 24 bytes on a 64-bit build, rounded up by the allocator to its size class. Ten million of those is around 240 MB, and they are invisible to `sys.getsizeof(readings)` because the list only points at them. Total: roughly 320–330 MB where the raw data is 80 MB — about four times over, entirely explained, with nothing leaking. Note also that identical readings are not shared. Small integers are cached by the interpreter; floats are not, so a million readings that all happen to equal 20.5 are a million distinct objects. CPython keeps a free list that makes *allocating* floats fast, but that does nothing for the footprint of live ones. ### The fix that is not a fix The reflex suggestion is to shorten the numbers: round each reading to two decimals before storing it. It saves nothing. A float object is a fixed-size struct whichever double it holds, so the rounded list occupies exactly as many bytes as the original — and the rounding has now introduced drift, because each stored reading is a slightly wrong value and any sum, mean or delta computed downstream carries accumulated error that the raw readings did not have. Paying in correctness for zero bytes is the worst trade in the incident. The same logic disposes of a few neighbours. Converting to `int` saves nothing structural for the same reason. Storing the numbers as short strings makes it *worse* — a small `str` object is larger than a float. Swapping the list for a tuple removes the over-allocation slack and shaves the header slightly, but the 240 MB of element objects is untouched, so it buys single-digit percent. ### What actually reduces it Ranked by how much they move the number: 1. **Do not materialize the buffer.** Most collectors do not need ten million readings resident; they need aggregates. Consuming an iterator and folding into running counts, sums, extrema or a fixed set of histogram buckets turns O(n) memory into O(1). This is the change that matters. 2. **Bound the window.** If recent history is genuinely required, keep a fixed-capacity window rather than everything since process start. An unbounded buffer is a leak in behaviour even when every byte is accounted for. 3. **Store the values, not objects wrapping them.** A compact fixed-width numeric buffer holds each reading in the eight bytes it actually needs, collapsing both terms into one 80 MB block and eliminating ten million object headers. The standard library's compact-buffer types are their own topic; the point here is that the saving comes from removing per-element objects, not from making the numbers smaller. 4. **Pre-size if the length is known.** `[None] * n` with index assignment, or building through the constructor from a sized iterable, avoids the resize copies and the transient peak where the old and new pointer arrays are both live. ### How to confirm it rather than guess `sys.getsizeof` on the list is the measurement that misleads here: it reports 80–90 MB against an RSS several times that, which is exactly the shape that makes people go hunting for a leak. Measure both terms — the list plus the sum over its elements — or, better, take `tracemalloc` snapshots and compare them, which attributes live allocations to the lines that made them and tells you whether the growth is your buffer or something else. Watch resident set size across a full collection cycle as well: a leak keeps climbing after the buffer is released, whereas a correctly sized buffer plateaus and drops. ### The judgement being tested The question is not really about floats. It is whether a candidate can reason about a Python container's real cost from its layout — pointers plus per-element objects — instead of assuming the memory model of a typed array, and whether they reach for the structural change (do not keep the data) before the cosmetic one (make the numbers shorter). Anyone who answers "10 million doubles should be 80 MB, so we have a leak" has assumed the wrong language.
- Would storing the readings in a tuple instead of a list help?Only marginally. A tuple is built at its exact length, so it carries no over-allocated slack, and its header is slightly smaller — that removes the eighth or so of waste on the pointer array. The 240 MB of float objects is untouched, because a tuple also stores pointers. It buys single-digit percent, which is not the order of magnitude the situation calls for.
- How would you confirm the memory is the buffer and not a leak?Measure both terms rather than trusting `sys.getsizeof` on the list, which reports only the pointer array and so looks far too small next to the process's resident set. Take `tracemalloc` snapshots at two points in a collection cycle and compare them — that attributes live allocations to the lines that made them. A correctly sized buffer plateaus and drops when released; a leak keeps climbing across cycles.
saying these in an interview costs you the question
- Treats sys.getsizeof on the list as its full memory cost
- Says rounding the readings to fewer decimals shrinks the list
- Assumes 10 million doubles must be 80 MB, so it is a leak
- Believes Python floats are bare machine doubles with no object overhead
- Blames over-allocation for the bulk of the overhead
- Expects equal float values to be shared like small integers