{"record":{"id":"5d1ed263ee3d51ae","repo":"astral-sh/ruff","slug":"stream-did-not-contain-valid-utf-8","errorCode":null,"errorMessage":"stream did not contain valid UTF-8","messagePattern":"stream did not contain valid UTF-8","errorType":"error_code","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"crates/ruff_db/src/system/memory_fs.rs","lineNumber":473,"sourceCode":"}\n\n#[derive(Debug)]\nstruct File {\n    content: Box<[u8]>,\n    last_modified: FileTime,\n}\n\n#[derive(Debug)]\nstruct Directory {\n    last_modified: FileTime,\n}\n\nfn not_found() -> std::io::Error {\n    std::io::Error::new(std::io::ErrorKind::NotFound, \"No such file or directory\")\n}\n\nfn invalid_utf8() -> std::io::Error {\n    std::io::Error::new(\n        std::io::ErrorKind::InvalidData,\n        \"stream did not contain valid UTF-8\",\n    )\n}\n\nfn create_dir_all(\n    paths: &mut RwLockWriteGuard<BTreeMap<Utf8PathBuf, Entry>>,\n    normalized: &Utf8Path,\n) -> Result<()> {\n    let mut path = Utf8PathBuf::new();\n\n    for component in normalized.components() {\n        path.push(component);\n        let mut inserted = false;\n        let entry = paths.entry(path.clone()).or_insert_with(|| {\n            inserted = true;\n            Entry::Directory(Directory {\n                last_modified: file_time_now(),","sourceCodeStart":455,"sourceCodeEnd":491,"githubUrl":"https://github.com/astral-sh/ruff/blob/26f38c119cac42e4d320ba08f09224fdec74af2c/crates/ruff_db/src/system/memory_fs.rs#L455-L491","documentation":"`invalid_utf8()` builds the memory file system's io::Error (kind InvalidData, 'stream did not contain valid UTF-8') used when reading stored file contents as a `String`. The memory FS stores raw bytes; `read_to_string` and `read_virtual_path_to_string` reject any file whose bytes are not valid UTF-8 with this error, matching `std::fs::read_to_string` semantics.","triggerScenarios":"Calling `read_to_string` or `read_virtual_path_to_string` on a memory-FS file whose contents were written with non-UTF-8 bytes (e.g. binary fixtures, latin-1 encoded sources).","commonSituations":"Tests storing binary or legacy-encoded fixtures then reading them as text, data generated by other tools with non-UTF-8 output, forgetting to use the byte-level read API.","solutions":["Write UTF-8 content into the memory FS fixture, or convert at write time","Use a byte-level read (`read_to_bytes`/equivalent) and decode explicitly with `String::from_utf8_lossy` if lossy text is acceptable","Detect the file's encoding before choosing the string read API"],"exampleFix":"// before\nlet text = fs.read_to_string(path)?; // fails on binary fixture\n// after\nlet bytes = fs.read(path)?;\nlet text = String::from_utf8_lossy(&bytes).into_owned();","handlingStrategy":"validation","validationCode":"let bytes = fs.read(path)?;\nif std::str::from_utf8(&bytes).is_err() {\n    return Err(io::Error::new(io::ErrorKind::InvalidData, \"fixture is not UTF-8\"));\n}","typeGuard":"fn is_utf8_file(fs: &MemoryFileSystem, path: &VfsPath) -> bool {\n    fs.read(path).map(|b| std::str::from_utf8(&b).is_ok()).unwrap_or(false)\n}","tryCatchPattern":"match fs.read_to_string(path) {\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => eprintln!(\"file is not UTF-8; use byte read\"),\n    Err(e) => return Err(e),\n    Ok(text) => text,\n}","preventionTips":["Store only UTF-8 text in memory-FS fixtures; encode binary data as base64 in test setup","Use byte-level reads when file contents may be binary","Validate fixture encodings in test helper constructors"],"tags":["filesystem","memory-fs","utf8","encoding"],"backgroundTag":"invalid-utf8-argument","analyzedSha":"26f38c119cac42e4d320ba08f09224fdec74af2c","analyzedAt":"2026-09-05T10:32:37.492Z","contentChangedAt":"2026-09-05T10:32:37.492Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}