On a Linux server, /dev/sdb is one of three physical volumes in an LVM volume group serving live databases, and it has started logging SMART errors and I/O timeouts. How do you get the data off that disk and remove it from the volume group without downtime?

## Assess before you move Two facts decide whether this is a routine evacuation or an incident: ``` pvs -o +pv_used # allocated versus free per physical volume vgs # VFree for the group as a whole lvs -o +devices vg0 # which logical volumes have extents on /dev/sdb ``` You need free extents on the *remaining* physical volumes at least equal to the allocated extents on `/dev/sdb`. Group-wide `VFree` is not sufficient on its own — free space that lives on the dying disk cannot receive its own data. If there is not enough room, add a replacement disk to the group first (`pvcreate` then `vgextend`) and evacuate onto it. That is usually the cleaner move anyway, since it keeps the group's total capacity constant. `lvs -o +devices` also tells you the blast radius: which services are exposed if the disk dies mid-evacuation. ## What pvmove actually does `pvmove /dev/sdb` relocates every allocated extent on that physical volume to free extents elsewhere in the group. Under the hood LVM converts each affected segment into a temporary mirror, synchronises the copy onto the destination extents, then atomically switches the logical volume's mapping to the new location and drops the old leg. Because the switch happens in the device-mapper table, the logical volume never goes away and the filesystem on it never notices — this is a genuinely online operation on mounted, busy volumes. Useful forms: ``` pvmove /dev/sdb # everything, LVM picks destinations pvmove /dev/sdb /dev/sdd # everything, onto a named destination pvmove -n data /dev/sdb # only the extents belonging to LV 'data' pvmove -i 10 /dev/sdb # report progress every 10 seconds ``` Progress is recorded in LVM metadata, not just in the running process, which is what makes it restartable. If the terminal dies, the machine reboots, or you stop it deliberately, running `pvmove` with no arguments resumes the outstanding move; `pvmove --abort` cancels it and leaves the extents where they started. Nothing is left half-mapped either way. ## The cost, which is real Every allocated extent is read from the failing disk and written elsewhere. On a multi-terabyte volume that is hours, and it is sequential-ish bulk I/O contending with production traffic for the same devices and the same queues. Watch service latency while it runs, and schedule it like the bulk copy it is. `-n` lets you evacuate the most critical logical volume first, so the exposure window for the data you care about most is short even if the whole disk takes all night. ## The failure case: reads that error pvmove's fundamental requirement is that it can *read* every extent on the disk it is emptying. A disk with reallocated sectors and rising timeouts may still satisfy that; a disk returning hard read errors will not, and the move will fail partway. That is the moment to stop optimising for elegance: - If the volumes are mirrored or RAID-backed below LVM, the redundancy — not pvmove — supplies the good copy, and the repair happens at that layer. - Otherwise, restore the affected logical volumes from backup onto healthy extents. And if the disk is already gone, evacuation is no longer on the table. LVM then shows the physical volume as missing, and `vgreduce --removemissing` (with `--force` when volumes still reference it) drops it from the group — at the cost of the logical volumes whose extents lived there, which then have to be recreated and restored. Knowing that this command sacrifices data is the difference between a controlled recovery and a second outage. ## Finishing the job Once `pvs` shows zero used extents on the disk: ``` vgreduce vg0 /dev/sdb # remove it from the volume group pvremove /dev/sdb # wipe the LVM label so it is no longer a PV ``` The order matters and is easy to reason about: `vgreduce` is group membership, `pvremove` is the label on the device. Trying `pvremove` on a device that is still a group member is refused. After that the disk can be pulled, and it is worth confirming with `pvs`/`vgs` that the group is healthy and that no logical volume lost extents. ## Say the whole arc in an interview The answer that lands is the ordered one: assess capacity and exposure, ensure a destination exists, evacuate online with pvmove while acknowledging its runtime cost, verify the disk is empty, remove it from the group, clear its label — and name the condition under which the plan changes to a restore, because a failing disk is not guaranteed to be a readable one.