In pandas 3.0, why does writing into the array from `df.to_numpy()` raise 'assignment destination is read-only', and how do you fix it?
answer
- the array shares the frame's memory
- writeable flag set to False
- one block versus mixed dtypes
- copy=True or the flag
basics
~10 sIn pandas 3.0, when the array shares memory with the frame, pandas returns a NumPy view with its writeable flag off so writes cannot bypass Copy-on-Write. Use to_numpy(copy=True) for a writeable copy.
solid answer
~40 sIn pandas 3.0 `to_numpy()`, `.values` and `np.asarray(df)` return a view when the array can share memory with the pandas object, as for a Series or a single-dtype frame, and pandas marks that view read-only. Writing into it would change the frame behind Copy-on-Write, so NumPy raises `ValueError: assignment destination is read-only`. A frame with mixed dtypes gets a newly built array, which is writeable, which is why the error can come and go. The fix is `df.to_numpy(copy=True)` or `arr.copy()`. Setting `arr.flags.writeable = True` also works and avoids the copy, but then writes reach the frame, so it is only safe when the frame is no longer needed.
code
python · 17 linesimport pandas as pd
scores = pd.DataFrame({"q1": [3, 5], "q2": [4, 2]}) # one int64 block
arr = scores.to_numpy()
print(arr.flags.writeable) # False: arr shares the frame's memory
try:
arr[0, 0] = 100
except ValueError as err:
print(err) # assignment destination is read-only
safe = scores.to_numpy(copy=True) # independent, writeable copy
safe[0, 0] = 100
print(scores.iloc[0, 0]) # 3: the frame is untouched
mixed = pd.DataFrame({"q1": [3, 5], "rate": [0.5, 0.7]})
print(mixed.to_numpy().flags.writeable) # True: mixed dtypes build a new arraygo deeper
Recall that in pandas 3.0 an array from to_numpy() may be read-only, and that to_numpy(copy=True) gives you one you can change.
Explain that the flag is cleared only when the array shares the frame's memory, which is why single-dtype frames and Series hit it and mixed frames do not.
Diagnose the error to the right layer, pandas sets the flag and NumPy raises, and choose a copy over flipping the flag unless profiling justifies it.
Set a team rule for code that hands frames to array routines that write in place: copy at the boundary, and allow the writeable flag only with a stated reason.
## What you see After an upgrade, a step that tweaks values through NumPy starts failing: ```python arr = scores.to_numpy() arr[0, 0] = 100 # ValueError: assignment destination is read-only ``` The error text comes from **NumPy**, but the cause is a **pandas 3.0** decision. When the array `to_numpy()` returns shares memory with the pandas object, pandas hands it back as a NumPy view whose **`writeable` flag is set to `False`**. NumPy then refuses any write into it. ## Why pandas 3.0 does this Copy-on-Write promises that one object can only be changed by modifying that object. A writeable NumPy view onto a frame's data would break the promise: writing into the array would change the frame, and any other object sharing the same data, behind pandas' back, without the copy that keeps objects separate. In 2.x that is exactly what happened, and it was a common way for data to change unexpectedly. In pandas 3.0 the user guide calls the protection "read-only NumPy arrays": the array "is set to non-writeable to protect against this behavior". ## When the array is read-only and when it is not Whether you get a read-only array depends on whether pandas could avoid a copy: | Object | What `to_numpy()` returns | Writeable? | |---|---|---| | A NumPy-backed Series | a view of its data | no | | A DataFrame whose columns live in one NumPy array, e.g. all `int64` | a view of that array | no | | A DataFrame with columns of different dtypes, e.g. `int64` and `float64` | a newly built array | yes | | Any of the above with `to_numpy(copy=True)` | a copy | yes | The same rule applies to the other ways of getting at the data: `df.values`, `Series.values` for NumPy-backed data, and `np.asarray(df)` all return read-only views when they share memory. This is also why the error can seem to come and go: adding a float column to an all-integer frame makes `to_numpy()` build a new, writeable array, and removing it brings the read-only view back. ## The two fixes The pandas user guide gives two options, and they are not equivalent: 1. **Copy.** Call `df.to_numpy(copy=True)`, or `arr.copy()` on the read-only view. You get an independent, writeable array, and the frame is untouched by anything you do to it. This is the right default for code that modifies an array in place, including any routine that writes into its input. 2. **Make the array writeable.** Setting `arr.flags.writeable = True` avoids the copy and is faster, but it **circumvents Copy-on-Write**: writes now go into memory the frame, and any object sharing it, still uses. The guide says to use it with caution, and it is only safe when you no longer care about the pandas object. A third option is often better than either: do the change in pandas, with a single assignment on the frame or a method like `where` or `clip`, and only convert at the end. ## Getting the layer right in an interview - **pandas** decides to return a view and sets the flag; that is the 3.0 behaviour change. - **NumPy** enforces the flag and raises `ValueError: assignment destination is read-only`. - The behaviour has nothing to do with `ChainedAssignmentError`, which is a pandas warning about indexing chains, and nothing to do with the removed `SettingWithCopyWarning`. A strong answer names the trigger (the array shares memory with the frame), the version (pandas 3.0, under Copy-on-Write), and picks the copy unless there is a measured reason to pay for the risk of the writeable flag. ## Where it bites in practice The error usually appears in older helpers that used NumPy for speed. A typical 2.x pattern centres a numeric frame in place: - `arr = features.values` followed by `arr -= arr.mean(axis=0)`, which relied on the write reaching `features`; - a loop that writes cleaned values into `series.to_numpy()` one position at a time; - a routine that receives `df.to_numpy()` and normalises it in place before returning nothing. In pandas 3.0 each of these raises the read-only `ValueError` when the frame is a single dtype, and code that relied on the write reaching the frame needs rewriting anyway, because Copy-on-Write no longer allows it. The durable fix is to express the step in pandas and rebind the name, for example `features = features - features.mean()`, and to give array routines a copy.
- What happens if you set `arr.flags.writeable = True` on `arr = s.to_numpy()` and then write into `arr`?The write succeeds and, because `arr` is a view of the Series' data, it changes `s` as well, and any object sharing that data. That bypasses the Copy-on-Write guarantee pandas 3.0 relies on, so the user guide describes it as faster but to be used with caution, only when you no longer care about the pandas object.
- Why might `df.to_numpy()` be writeable on one frame and read-only on another in pandas 3.0?Because the flag is only cleared when the array shares memory with the frame. A frame whose columns all live in one NumPy array, such as all `int64`, can be returned as a view, which is read-only. A frame with several dtypes has to be combined into a new array, which shares nothing and is writeable.
- Every array from to_numpy() is read-only in pandas 3.0, even a fresh one
- pandas raises ChainedAssignmentError when you write to the array
- Setting writeable=True is a free fix with no effect on the frame
- The read-only error means the array was copied from the frame
- Passing copy=False to to_numpy makes the view writeable again
Assumes pandas 3.0; in 2.x to_numpy() could return a writeable view whose edits changed the frame, in 3.0 a view that shares memory is read-only.