A Ruby job importing a legacy Latin-1 customer file raises ArgumentError: invalid byte sequence in UTF-8 on split; how do you diagnose and fix it?

## What actually went wrong A Ruby `String` is bytes plus an `Encoding` label. When an importer reads a file without saying what encoding it is in, Ruby labels the result with `Encoding.default_external`, which on a typical server is UTF-8. The legacy system wrote **ISO-8859-1** (Latin-1), where `é` is the single byte `0xE9`. In UTF-8, `0xE9` must start a three-byte sequence, so a lone `0xE9` makes the string **invalid**, but nothing checks that at read time. The failure surfaces later, in whichever method first needs to decode characters: - `String#split` and every Regexp match (`=~`, `match?`, `gsub` with a Regexp) raise `ArgumentError: invalid byte sequence in UTF-8`. - Other methods may quietly succeed, so the crash appears far from the read, often on the one customer whose name has an accent. ## Diagnosing on one failing row 1. **Reproduce with the exact row** that fails, not the whole file. 2. **Ask Ruby what it believes**: `row.encoding` shows the label, `row.valid_encoding?` returns `false`. 3. **Look at the bytes**: `row.bytes.find { it > 127 }` returning `233` (0xE9) next to ASCII letters is the signature of Latin-1 `é`. 4. **Confirm with the producer** which encoding the export uses. Ruby does not guess encodings; `valid_encoding?` only says the bytes do not fit the label they carry. ```ruby line = File.foreach("customers_legacy.txt").first line.encoding # => #<Encoding:UTF-8> (default_external) line.valid_encoding? # => false line.bytes.find { it > 127 } # => 233, a lone 0xE9 utf8 = line.encode("UTF-8", "ISO-8859-1") utf8.valid_encoding? # => true utf8.split(";") # no ArgumentError ``` ## Fixing it at the boundary - **Transcode, do not relabel to UTF-8.** `encode("UTF-8", "ISO-8859-1")` reads the bytes as Latin-1 and writes proper UTF-8. `force_encoding("UTF-8")` changes nothing, because the label already says UTF-8. - **Do it once, where data enters.** Either pass the external encoding when opening the file (the IO layer can transcode as it reads) or convert each record immediately after reading. Everything downstream then sees valid UTF-8 only. - **Expect no conversion errors from Latin-1.** Every one of the 256 byte values has a character in ISO-8859-1, so `encode` from it to UTF-8 does not raise `Encoding::UndefinedConversionError` or `Encoding::InvalidByteSequenceError`. - **Verify after converting.** Assert `valid_encoding?` on output and count converted rows in the job's log. ## Traps a senior engineer checks for | Symptom after the fix | Likely cause | Response | |---|---|---| | `“` and `€` show up as invisible control characters | producer wrote **Windows-1252**, not Latin-1 | use `encode("UTF-8", "Windows-1252")` | | `é` becomes `é` | a row was already UTF-8 and got converted again | convert only rows whose `valid_encoding?` is false | | accents replaced by `�` | someone added `scrub` | remove it; the bytes were never corrupt | Windows-1252 and ISO-8859-1 agree on `0xE9` but differ in `0x80`–`0x9F`: Windows-1252 puts curly quotes, dashes and the euro sign there, while ISO-8859-1 has C1 control characters. Characters U+0080–U+009F in converted output are the tell. For a file that mixes old Latin-1 rows with newer UTF-8 rows, a per-row rule works: keep rows that are already `valid_encoding?`, transcode the rest. It is a heuristic, since some Latin-1 byte pairs happen to form valid UTF-8, so prefer fixing the producer or splitting the file by origin. ## Preventing a recurrence 1. **Write the source encoding into the importer's configuration**, next to the file path, rather than relying on whatever `default_external` the server happens to have. 2. **Add a fixture row with accented names** (`José`, `Müller`, `Zoë`) to the importer's tests; ASCII-only fixtures are why this bug reached production. 3. **Validate at the boundary**: reject or quarantine rows whose converted text still fails `valid_encoding?`, and count them in the job's output. 4. **Log the row identifier and the offending byte**, never the whole customer record, so the next failure is diagnosable without leaking personal data. ## Why scrub is the wrong default `scrub` replaces each invalid sequence with a replacement string, U+FFFD for a Unicode encoding. It is the right tool for input that is genuinely corrupt, such as a truncated upload, where you accept losing the damaged bytes. Here the bytes were valid Latin-1 all along; scrubbing turns `José` into `Jos�` and destroys customer data silently. Use it only after transcoding, on rows you have decided to accept with losses, and log each one.