astral-sh/ruff · error · io::Error

stream did not contain valid UTF-8

Error message

stream did not contain valid UTF-8

What it means

`invalid_utf8()` builds the memory file system's io::Error (kind InvalidData, 'stream did not contain valid UTF-8') used when reading stored file contents as a `String`. The memory FS stores raw bytes; `read_to_string` and `read_virtual_path_to_string` reject any file whose bytes are not valid UTF-8 with this error, matching `std::fs::read_to_string` semantics.

Source

Thrown at crates/ruff_db/src/system/memory_fs.rs:473

}

#[derive(Debug)]
struct File {
    content: Box<[u8]>,
    last_modified: FileTime,
}

#[derive(Debug)]
struct Directory {
    last_modified: FileTime,
}

fn not_found() -> std::io::Error {
    std::io::Error::new(std::io::ErrorKind::NotFound, "No such file or directory")
}

fn invalid_utf8() -> std::io::Error {
    std::io::Error::new(
        std::io::ErrorKind::InvalidData,
        "stream did not contain valid UTF-8",
    )
}

fn create_dir_all(
    paths: &mut RwLockWriteGuard<BTreeMap<Utf8PathBuf, Entry>>,
    normalized: &Utf8Path,
) -> Result<()> {
    let mut path = Utf8PathBuf::new();

    for component in normalized.components() {
        path.push(component);
        let mut inserted = false;
        let entry = paths.entry(path.clone()).or_insert_with(|| {
            inserted = true;
            Entry::Directory(Directory {
                last_modified: file_time_now(),

View on GitHub (pinned to 26f38c119c)

Solutions

1. Write UTF-8 content into the memory FS fixture, or convert at write time
2. Use a byte-level read (`read_to_bytes`/equivalent) and decode explicitly with `String::from_utf8_lossy` if lossy text is acceptable
3. Detect the file's encoding before choosing the string read API

Example fix

// before
let text = fs.read_to_string(path)?; // fails on binary fixture
// after
let bytes = fs.read(path)?;
let text = String::from_utf8_lossy(&bytes).into_owned();

let bytes = fs.read(path)?;
if std::str::from_utf8(&bytes).is_err() {
    return Err(io::Error::new(io::ErrorKind::InvalidData, "fixture is not UTF-8"));
}

fn is_utf8_file(fs: &MemoryFileSystem, path: &VfsPath) -> bool {
    fs.read(path).map(|b| std::str::from_utf8(&b).is_ok()).unwrap_or(false)
}

match fs.read_to_string(path) {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => eprintln!("file is not UTF-8; use byte read"),
    Err(e) => return Err(e),
    Ok(text) => text,
}

When it happens

Trigger: Calling `read_to_string` or `read_virtual_path_to_string` on a memory-FS file whose contents were written with non-UTF-8 bytes (e.g. binary fixtures, latin-1 encoded sources).

Common situations: Tests storing binary or legacy-encoded fixtures then reading them as text, data generated by other tools with non-UTF-8 output, forgetting to use the byte-level read API.

Related errors

• InvalidUtf8
• Invalid UTF-8 in JUnit report
• File already exists
• No such file or directory
• NotFound

AI-assisted analysis of astral-sh/ruff@26f38c119c (2026-09-05). Data as JSON: /api/errors/5d1ed263ee3d51ae. Report an issue: GitHub.