{"record":{"id":"a7b9a50de42a4ab5","repo":"microsoft/edit","slug":"stream-did-not-contain-valid-utf-8","errorCode":null,"errorMessage":"stream did not contain valid UTF-8","messagePattern":"stream did not contain valid UTF-8","errorType":"error_code","errorClass":"io::Error (ErrorKind::InvalidData)","httpStatus":null,"severity":"error","filePath":"crates/stdext/src/arena/fs.rs","lineNumber":47,"sourceCode":"                Ok(n) => {\n                    unsafe { vec.set_len(vec.len() + n) };\n                    buf_size = (buf_size * 2).min(MAX_SIZE);\n                }\n                Err(e) if e.kind() == io::ErrorKind::Interrupted => {}\n                Err(e) => return Err(e),\n            }\n        }\n\n        Ok(vec)\n    }\n    inner(arena, path.as_ref())\n}\n\npub fn read_to_string<P: AsRef<Path>>(arena: &Arena, path: P) -> io::Result<BString<'_>> {\n    fn inner<'a>(arena: &'a Arena, path: &Path) -> io::Result<BString<'a>> {\n        let vec = read_to_vec(arena, path)?;\n        BString::from_utf8(vec).map_err(|_| {\n            io::Error::new(io::ErrorKind::InvalidData, \"stream did not contain valid UTF-8\")\n        })\n    }\n    inner(arena, path.as_ref())\n}\n\nfn file_read_uninit<T: Read>(file: &mut T, buf: &mut [MaybeUninit<u8>]) -> io::Result<usize> {\n    unsafe {\n        let buf_slice = from_raw_parts_mut(buf.as_mut_ptr().cast(), buf.len());\n        let n = file.read(buf_slice)?;\n        Ok(n)\n    }\n}\n","sourceCodeStart":29,"sourceCodeEnd":60,"githubUrl":"https://github.com/microsoft/edit/blob/826b4c097b6f14ba0a846dc56f2f0223a3aaf73a/crates/stdext/src/arena/fs.rs#L29-L60","documentation":"read_to_string in crates/stdext/src/arena/fs.rs reads the file at `path` into a byte vector from the arena, then attempts to convert it to a UTF-8 BString. If the bytes are not valid UTF-8, the conversion error is mapped to an io::Error of kind InvalidData with the message 'stream did not contain valid UTF-8'. It mirrors std::io::read_to_string semantics: the function only guarantees valid string output, so invalid encodings are rejected.","triggerScenarios":"Calling `stdext::arena::fs::read_to_string(&arena, path)` on a file whose bytes are not valid UTF-8 — e.g. a binary file, a Latin-1/Windows-1252 or UTF-16 text file, or a file truncated mid multi-byte UTF-8 sequence.","commonSituations":"Reading config or data files saved by other tools in a non-UTF-8 locale encoding; accidentally pointing at binary artifacts (images, gzip, sqlite) instead of text; network-downloaded content with a BOM or different encoding; corrupted/partially-written files.","solutions":["Check the file's actual encoding (`file <path>` or `chardet`) and convert it to UTF-8 (`iconv -f LATIN1 -t UTF-8 in > out`) before reading","If you only need raw bytes, use `read_to_vec` (the underlying helper) instead of read_to_string, avoiding the UTF-8 requirement","If you need lossy text, decode the bytes yourself with String::from_utf8_lossy after read_to_vec","If truncation is suspected, re-obtain the file and verify integrity (size/checksum) before reading"],"exampleFix":"// before\nlet s = stdext::arena::fs::read_to_string(&arena, path)?;\n// after (lossy fallback)\nlet bytes = stdext::arena::fs::read_to_vec(&arena, path)?;\nlet s = String::from_utf8(bytes.to_vec())\n    .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, \"stream did not contain valid UTF-8\"))\n    .unwrap_or_else(|_| String::from_utf8_lossy(&bytes).into_owned());","handlingStrategy":"validation","validationCode":"fn is_valid_utf8(arena: &Arena, path: &Path) -> bool {\n    match stdext::arena::fs::read_to_vec(arena, path) {\n        Ok(v) => std::str::from_utf8(&v).is_ok(),\n        Err(_) => false,\n    }\n}","typeGuard":"fn as_utf8(bytes: &[u8]) -> Option<&str> {\n    std::str::from_utf8(bytes).ok()\n}","tryCatchPattern":"match stdext::arena::fs::read_to_string(&arena, path) {\n    Ok(s) => { /* use s */ }\n    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains(\"valid UTF-8\") => {\n        // fall back to read_to_vec + from_utf8_lossy, or reject the input file\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Detect file encoding before parsing (BOM sniffing, chardet, or `file` command) and transcode non-UTF-8 inputs to UTF-8 upstream","Use read_to_vec plus String::from_utf8_lossy when the input may contain arbitrary bytes","Validate checksums/sizes of downloaded files to catch truncation that splits multi-byte sequences","Keep text artifacts in UTF-8 across toolchains and CI environments (set locale, editor encoding)"],"tags":["io","utf-8","encoding","filesystem"],"backgroundTag":"file-read-failed","analyzedSha":"826b4c097b6f14ba0a846dc56f2f0223a3aaf73a","analyzedAt":"2026-09-06T13:30:05.543Z","contentChangedAt":"2026-09-06T13:30:05.543Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}