skip to content

In Rust, how do `String::from_utf8`, `String::from_utf8_lossy` and `String::from_utf8_unchecked` differ when turning bytes read from a socket into text?

level: middleimportance: should knowfreq 45%

answer

  1. validate, replace, or trust
  2. Result versus Cow versus unsafe
  3. U+FFFD replacement character
  4. error_len None means truncated input

basics

~10 s

from_utf8 validates a Vec<u8> and returns Result<String, FromUtf8Error>. from_utf8_lossy replaces invalid sequences with U+FFFD and returns Cow<str>. The unsafe from_utf8_unchecked skips the check, and invalid bytes then break every later string operation's assumption.

solid answer

~40 s

`String::from_utf8(vec)` checks the bytes and, on success, takes over the vector's buffer without copying; on failure the `FromUtf8Error` gives the vector back through `into_bytes()` and says where decoding stopped through `utf8_error().valid_up_to()`. `str::from_utf8(&bytes)` does the same check on a borrowed slice and returns `&str`. `String::from_utf8_lossy(&bytes)` never fails: it replaces each invalid sequence with U+FFFD and returns a `Cow<str>` that borrows when the input was already valid. Rust 1.99 added `String::from_utf8_lossy_owned(vec)`, which returns a plain `String`. `from_utf8_unchecked` is `unsafe`: it skips validation, and the caller must guarantee valid UTF-8, because the rest of std assumes it. With a socket, remember that a character can be split across two reads.

code

rust · 22 lines
rust
fn main() {
    // "naïve " in UTF-8, then Latin-1 bytes that are not valid UTF-8.
    let frame: Vec<u8> = b"na\xC3\xAFve \xE9t\xE9".to_vec();

    match String::from_utf8(frame.clone()) {
        Ok(text) => println!("valid: {text}"),
        Err(e) => {
            let valid_up_to = e.utf8_error().valid_up_to();
            let bytes = e.into_bytes(); // the original Vec<u8>
            println!("invalid after byte {valid_up_to} of {}", bytes.len());
        }
    }

    let shown = String::from_utf8_lossy(&frame); // Cow<str>
    println!("lossy: {shown}");

    let owned: String = String::from_utf8_lossy_owned(frame); // stable since 1.99
    println!("owned: {owned} ({} bytes)", owned.len());

    let ok = std::str::from_utf8(b"plain ascii").unwrap(); // &str, no allocation
    println!("{ok}");
}

go deeper

for a junior

Recall that bytes become a String through from_utf8, which can fail, or from_utf8_lossy, which replaces bad bytes with a replacement character.

for a middle

Explain the three policies, the return types Result, Cow and unsafe, what FromUtf8Error gives back, and that from_utf8_lossy_owned arrived in Rust 1.99.

for a senior

Handle streaming input correctly: keep an incomplete tail when error_len is None, never lossy-decode chunks independently, and reject rather than repair text that will be parsed, compared or signed.

for a principal

Set the policy per boundary: strict decoding at protocol edges, lossy decoding only for display and logs, and from_utf8_unchecked behind a reviewed SAFETY comment.

## The invariant behind all of them A Rust `String` and `str` are always valid **UTF-8**, and every method on them (`chars`, slicing, `find`, formatting) relies on that. Bytes from outside the program, such as a socket, a file or a subprocess, carry no such promise. Converting them is therefore a choice among three policies: **validate and fail**, **validate and repair**, or **trust without checking**. | Function | Input | Returns | On invalid bytes | |---|---|---|---| | `String::from_utf8` | `Vec<u8>` | `Result<String, FromUtf8Error>` | `Err`, vector handed back | | `str::from_utf8` | `&[u8]` | `Result<&str, Utf8Error>` | `Err` | | `String::from_utf8_lossy` | `&[u8]` | `Cow<'_, str>` | each bad sequence becomes U+FFFD | | `String::from_utf8_lossy_owned` (1.99) | `Vec<u8>` | `String` | same replacement | | `String::from_utf8_unchecked` | `Vec<u8>` | `String` (an `unsafe fn`) | no check: the caller's responsibility | ## Validate and fail: `from_utf8` `String::from_utf8` checks the bytes and, if they are valid, reuses the vector's allocation; its documentation says it takes care not to copy the vector. If they are not, the **`FromUtf8Error`** keeps the vector, so nothing is lost: - `err.utf8_error().valid_up_to()` is the length of the valid prefix; - `err.utf8_error().error_len()` is `Some(n)` for an invalid sequence of `n` bytes, or `None` when the input simply **ended in the middle of a character**; - `err.into_bytes()` returns the original `Vec<u8>`. Use this when invalid text is an error in your protocol, for example a JSON body or a header that the specification says is UTF-8. `str::from_utf8` is the borrowed twin: it allocates nothing and returns a `&str` that points into your buffer. ## Validate and repair: `from_utf8_lossy` `String::from_utf8_lossy` never fails. It replaces every invalid sequence with **U+FFFD REPLACEMENT CHARACTER**, usually drawn as a question mark in a diamond. Its return type is a **`Cow<str>`**: when the input is already valid it borrows the bytes with no allocation, and only when replacement is needed does it build an owned `String`. Rust 1.99 stabilised **`from_utf8_lossy_owned`**, which takes a `Vec<u8>` and always returns a `String`, convenient when you own the bytes and want an owned result; its docs note that reusing the original allocation is not guaranteed. Use lossy decoding for display and logging, where showing something beats failing. Do not use it where the text is later compared, parsed or signed: replacement changes the data, and two different inputs can decode to the same string. ## Trust without checking: `from_utf8_unchecked` `String::from_utf8_unchecked` is an `unsafe fn`. Its safety section says it does not check that the bytes are valid UTF-8, and that violating this may cause memory unsafety issues with future users of the `String`, because the rest of the standard library assumes valid UTF-8. Note the precise scope: the Reference treats a `str` value like `[u8]` for validity, so creating the string is not itself listed as undefined behaviour, but later std code that decodes it can misbehave, up to memory unsafety. Use it only when validity is already proven, for example bytes you just produced from `char`s, and write a `// SAFETY:` comment saying why. ## The socket trap: characters split across reads A network read returns however many bytes arrived. A multi-byte character can be cut in half between two reads: 1. Decoding each chunk with `from_utf8` reports an error whose `error_len()` is `None`; the docs say this could be a valid character spanning chunks. 2. Decoding each chunk with `from_utf8_lossy` silently turns both halves into U+FFFD, corrupting valid text. 3. The fix is to keep the incomplete tail, at most three bytes and starting at `valid_up_to()`, and prepend it to the next read, or to read the whole message first and decode once. A buffered reader that yields lines, such as `BufRead::read_line`, sidesteps the split because it reads up to a newline before validating, and its docs say it returns an error if the bytes read are not valid UTF-8. ## Choosing in practice | Situation | Use | |---|---| | a protocol field that must be UTF-8 | `String::from_utf8` or `str::from_utf8`, rejecting on `Err` | | untrusted bytes shown in a log or UI | `String::from_utf8_lossy` | | an owned `Vec<u8>` that should become a repaired `String` | `String::from_utf8_lossy_owned`, on Rust 1.99 or later | | bytes you just built from `char`s or valid `&str`s | the safe APIs first; `from_utf8_unchecked` only if profiling shows the check matters | | a stream read in chunks | carry the incomplete tail between reads, or decode whole lines or messages | The validation pass is a single linear scan, so the unsafe shortcut rarely repays the review burden it creates. And whichever function you pick, decide it once per boundary: the same bytes decoded strictly in one service and lossily in another produce two different strings.

  • Why does `from_utf8` take a `Vec<u8>` instead of `&[u8]`?
    So that on success it can reuse the vector's buffer as the `String`'s buffer without copying. On failure the `FromUtf8Error` returns the same vector through `into_bytes()`. When you only have a borrowed slice, `str::from_utf8` validates it and returns a `&str` without allocating.
  • What does `error_len()` returning `None` tell a streaming decoder?
    That the input ended partway through a multi-byte sequence rather than containing a bad byte. The bytes from `valid_up_to()` on are a possibly valid character split across reads, so keep them and prepend them to the next chunk instead of reporting an error or replacing them.
  • Is calling `from_utf8_unchecked` on invalid bytes immediately undefined behaviour?
    The Reference does not list it as such: a `str` only has to be initialised bytes to be a valid value. But it breaks the safety invariant every std string method relies on, and the docs warn of memory unsafety in later users, so treat it as a soundness bug either way.

  • from_utf8_lossy returns an error when the bytes are invalid.
  • String::from_utf8 copies the bytes into a new buffer.
  • from_utf8_unchecked is fine for socket data because a String checks itself later.
  • Lossy decoding each network chunk separately is safe for valid text.
  • A failed from_utf8 call drops the original byte vector.

Assumes Rust 1.99; String::from_utf8_lossy_owned is stable since Rust 1.99, while from_utf8, from_utf8_lossy and from_utf8_unchecked date from 1.0.