skip to content

A Rust backup tool panics at `path.to_str().unwrap()` on a Latin-1 file name; why, and how should it carry paths using `OsString` and `PathBuf`?

level: seniorimportance: should knowfreq 35%

answer

  1. file names are not always UTF-8
  2. OsStr bridges platform strings
  3. to_str returns an Option
  4. display() and args_os()

basics

~20 s

File names on Unix can be arbitrary non-zero bytes and on Windows arbitrary 16-bit units, while str must be UTF-8. Path::to_str returns None for such a name, so unwrap panics. Keep paths as &Path and PathBuf end to end, and convert only for display.

solid answer

~40 s

`OsString` and `OsStr` exist because operating-system strings need not be valid Unicode: Unix file names can be arbitrary non-zero bytes, Windows names arbitrary 16-bit values. `PathBuf` and `Path` wrap them, so every path API (`push`, `join`, `file_name`, `extension`, `set_extension`, `fs::copy`) works on a name like `caf\xE9.txt`. `to_str()` is the one step that demands Unicode: it returns `Option<&str>`, `None` here, and the `unwrap` panics. So keep `&Path` and `PathBuf` through the whole pipeline, print with `path.display()`, which is lossy, use `to_string_lossy()` when you need text and can accept U+FFFD, and use `into_string()` on an `OsString` when you need a `String` and want the original back on failure. Read arguments with `std::env::args_os()`: `std::env::args()` panics on a non-Unicode argument.

code

rust · 19 lines
rust
use std::ffi::OsStr;
use std::os::unix::ffi::OsStrExt; // Unix-only: an OsStr from raw bytes
use std::path::{Path, PathBuf};

fn main() {
    // A Latin-1 file name: 0xE9 is not valid UTF-8 on its own.
    let raw = OsStr::from_bytes(b"caf\xE9.txt");
    let path: &Path = Path::new(raw);

    println!("to_str: {:?}", path.to_str()); // None
    println!("lossy:  {}", path.to_string_lossy());
    println!("debug:  {path:?}");
    println!("ext:    {:?}", path.extension()); // Some("txt")

    let mut out = PathBuf::from("/srv/archive");
    out.push(path.file_name().unwrap()); // raw bytes kept
    out.set_extension("bak");
    println!("joined: {}", out.display());
}

go deeper

for a junior

Recall that file names may not be valid UTF-8, so Path::to_str returns an Option, and that path.display() is how you print a path.

for a middle

Explain OsString and OsStr as the bridge to platform strings, PathBuf and Path as wrappers, and which conversions fail, which are lossy and which return the original on error.

for a senior

Diagnose the crash and fix the pipeline: read with args_os, keep PathBuf through every file operation, convert lossily only for logs, and test with a non-UTF-8 name on Unix.

for a principal

Make paths-stay-paths a codebase rule for file tools, with String conversions allowed only at named edges such as UI or JSON output, where the lossy or failing behaviour is decided explicitly.

## Why `str` cannot hold every file name Rust's `str` is always valid **UTF-8**. Operating systems make weaker promises, and the `OsString` documentation spells them out: - on **Unix**, strings such as file names are often arbitrary sequences of non-zero bytes, in many cases interpreted as UTF-8; - on **Windows**, they are often arbitrary sequences of non-zero 16-bit values, interpreted as UTF-16 when valid. A file created years ago by a Latin-1 system can be named `caf\xE9.txt`, where byte `0xE9` is `é` in Latin-1 and invalid in UTF-8. A Windows name can contain an unpaired surrogate. Neither fits in a `String`. ## The bridge types | Owned | Borrowed | Holds | |---|---|---| | `String` | `&str` | valid UTF-8 only | | `OsString` | `&OsStr` | any platform string | | `PathBuf` | `&Path` | an `OsString` / `OsStr` with path methods | | `CString` | `&CStr` | NUL-terminated bytes for C APIs | `OsString` is to `&OsStr` what `String` is to `&str`. A `&str` converts to `&OsStr` and `&Path` for free, so text paths flow in easily; the reverse direction is the one that can fail. `OsString` is not NUL-terminated, which is why calls into C use `CString` instead, a separate topic covered with foreign function calls. ## Where the panic comes from Every `Path` method that returns a component keeps the platform form: `file_name()` and `extension()` return `Option<&OsStr>`, and `join`, `push` and `set_extension` build `PathBuf`s. The conversions to text are the only places Unicode is demanded: | Method | Returns | Non-Unicode input | |---|---|---| | `Path::to_str` / `OsStr::to_str` | `Option<&str>` | `None` | | `Path::to_string_lossy` | `Cow<'_, str>` | invalid parts become U+FFFD | | `OsString::into_string` | `Result<String, OsString>` | `Err` with the original value | | `Path::display` | a `Display` adapter | lossy output for printing | `path.to_str().unwrap()` turns that `None` into a panic. In a backup tool the first odd file name aborts the whole run, and if the code had used `to_string_lossy()` and then opened the result, it would ask for a file that does not exist and fail with a not-found error, because `caf�.txt` is a different name. ## Carrying paths correctly 1. **Read input as OS strings.** `std::env::args()` panics during iteration if any argument is not valid Unicode; `std::env::args_os()` yields `OsString`s. Directory listings already give `DirEntry::path()` as a `PathBuf`. 2. **Keep `&Path` and `PathBuf` through the pipeline.** Build destinations with `join` and `set_extension`, and pass them straight to `std::fs`. A function parameter of type `impl AsRef<Path>` accepts `&str`, `String`, `&Path` and `PathBuf` alike. 3. **Convert only at the edges.** Print with `path.display()` or `{:?}` (which shows `"caf\xE9.txt"` unambiguously); write logs with `to_string_lossy()`; demand a `String` with `into_string()` only where the format truly requires UTF-8, and handle the `Err`. 4. **Need the raw bytes on Unix?** `std::os::unix::ffi::OsStrExt` adds `as_bytes()` and `from_bytes()`. It is platform-specific, so keep it behind `#[cfg(unix)]`. ## What rustc does for you `PathBuf` deliberately does not implement `Display`. `format!("{out}")` fails with E0277, "`PathBuf` doesn't implement `std::fmt::Display`", and rustc 1.99 adds the note "call `.display()` or `.to_string_lossy()` to safely print paths, as they may contain non-Unicode data". The compiler is forcing exactly the decision this question is about: printing a path is lossy, and the code has to say so. ## Platform notes - **Windows storage.** The `OsString` docs say that on Windows the value is not stored as UTF-16 internally but as 8-bit values in a less strict variant of UTF-8, which is how it can hold a name containing an unpaired surrogate. `std::os::windows::ffi::OsStrExt::encode_wide` produces the 16-bit units a Windows API call needs. - **Capacity units.** For the same reason, `OsString::capacity` counts UTF-8 bytes for content created from valid Unicode and bytes in an unspecified encoding otherwise, so do not compare it with a `String`'s capacity. - **Testing.** A Unix test can build a non-UTF-8 name in memory with `OsStr::from_bytes`, as the example does, without touching the file system; some file systems refuse to store such names, so a test that creates real files may behave differently across machines.

  • Why does `std::env::args()` panic when `std::env::args_os()` does not?
    `args()` yields `String`s, and its docs say the iterator panics during iteration if any process argument is not valid Unicode. `args_os()` yields `OsString`s and accepts anything the OS passes. A tool that takes file names on the command line should read them with `args_os()` and turn them into `PathBuf`s.
  • When is `to_string_lossy()` the wrong conversion?
    Whenever the string is used to find the file again. Replacing an invalid byte with U+FFFD produces a different name, so opening, copying or comparing the lossy result targets a file that does not exist. Lossy conversion is for display and logs only; keep the `Path` for file operations.
  • How do you get the raw bytes of an `OsStr` on Linux?
    Import `std::os::unix::ffi::OsStrExt` and call `as_bytes()`; `OsStr::from_bytes` goes the other way. These exist only on Unix targets, so gate the code with `#[cfg(unix)]`. On Windows, `std::os::windows::ffi::OsStrExt::encode_wide` gives the 16-bit units instead.

  • File names are always valid UTF-8 on modern systems, so to_str().unwrap() is safe.
  • Converting with to_string_lossy() and opening the result finds the same file.
  • PathBuf is a String with extra methods, so it implements Display.
  • std::env::args() skips arguments that are not valid Unicode.
  • OsString is NUL-terminated and can be passed straight to C functions.