microsoft/edit · error · io::Error (ErrorKind::InvalidData)

stream did not contain valid UTF-8

Error message

stream did not contain valid UTF-8

What it means

read_to_string in crates/stdext/src/arena/fs.rs reads the file at `path` into a byte vector from the arena, then attempts to convert it to a UTF-8 BString. If the bytes are not valid UTF-8, the conversion error is mapped to an io::Error of kind InvalidData with the message 'stream did not contain valid UTF-8'. It mirrors std::io::read_to_string semantics: the function only guarantees valid string output, so invalid encodings are rejected.

Source

Thrown at crates/stdext/src/arena/fs.rs:47

                Ok(n) => {
                    unsafe { vec.set_len(vec.len() + n) };
                    buf_size = (buf_size * 2).min(MAX_SIZE);
                }
                Err(e) if e.kind() == io::ErrorKind::Interrupted => {}
                Err(e) => return Err(e),
            }
        }

        Ok(vec)
    }
    inner(arena, path.as_ref())
}

pub fn read_to_string<P: AsRef<Path>>(arena: &Arena, path: P) -> io::Result<BString<'_>> {
    fn inner<'a>(arena: &'a Arena, path: &Path) -> io::Result<BString<'a>> {
        let vec = read_to_vec(arena, path)?;
        BString::from_utf8(vec).map_err(|_| {
            io::Error::new(io::ErrorKind::InvalidData, "stream did not contain valid UTF-8")
        })
    }
    inner(arena, path.as_ref())
}

fn file_read_uninit<T: Read>(file: &mut T, buf: &mut [MaybeUninit<u8>]) -> io::Result<usize> {
    unsafe {
        let buf_slice = from_raw_parts_mut(buf.as_mut_ptr().cast(), buf.len());
        let n = file.read(buf_slice)?;
        Ok(n)
    }
}

View on GitHub (pinned to 826b4c097b)

Solutions

1. Check the file's actual encoding (`file <path>` or `chardet`) and convert it to UTF-8 (`iconv -f LATIN1 -t UTF-8 in > out`) before reading
2. If you only need raw bytes, use `read_to_vec` (the underlying helper) instead of read_to_string, avoiding the UTF-8 requirement
3. If you need lossy text, decode the bytes yourself with String::from_utf8_lossy after read_to_vec
4. If truncation is suspected, re-obtain the file and verify integrity (size/checksum) before reading

Example fix

// before
let s = stdext::arena::fs::read_to_string(&arena, path)?;
// after (lossy fallback)
let bytes = stdext::arena::fs::read_to_vec(&arena, path)?;
let s = String::from_utf8(bytes.to_vec())
    .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "stream did not contain valid UTF-8"))
    .unwrap_or_else(|_| String::from_utf8_lossy(&bytes).into_owned());

fn is_valid_utf8(arena: &Arena, path: &Path) -> bool {
    match stdext::arena::fs::read_to_vec(arena, path) {
        Ok(v) => std::str::from_utf8(&v).is_ok(),
        Err(_) => false,
    }
}

fn as_utf8(bytes: &[u8]) -> Option<&str> {
    std::str::from_utf8(bytes).ok()
}

match stdext::arena::fs::read_to_string(&arena, path) {
    Ok(s) => { /* use s */ }
    Err(e) if e.kind() == io::ErrorKind::InvalidData && e.to_string().contains("valid UTF-8") => {
        // fall back to read_to_vec + from_utf8_lossy, or reject the input file
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling `stdext::arena::fs::read_to_string(&arena, path)` on a file whose bytes are not valid UTF-8 — e.g. a binary file, a Latin-1/Windows-1252 or UTF-16 text file, or a file truncated mid multi-byte UTF-8 sequence.

Common situations: Reading config or data files saved by other tools in a non-UTF-8 locale encoding; accidentally pointing at binary artifacts (images, gzip, sqlite) instead of text; network-downloaded content with a BOM or different encoding; corrupted/partially-written files.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

AI-assisted analysis of microsoft/edit@826b4c097b (2026-09-06). Data as JSON: /api/errors/a7b9a50de42a4ab5. Report an issue: GitHub.