Hmbown/CodeWhale · error · std::io::Error

stream did not contain valid UTF-8

Error message

stream did not contain valid UTF-8

What it means

InvalidData with std's classic message, raised by the file-reading tool in tools/file.rs when any line read from the file is not valid UTF-8. The windowed reader validates every line exactly so behavior matches the old whole-file read_to_string: one invalid byte anywhere in the file fails the read, not just in the requested window.

Source

Thrown at crates/tui/src/tools/file.rs:1042

    loop {
        raw.clear();
        let n = reader.read_until(b'\n', &mut raw)?;
        if n == 0 {
            break;
        }
        // Mirror `str::lines`: strip the trailing '\n', and a '\r' only when
        // it directly precedes that '\n'.
        let mut end = raw.len();
        if raw[..end].ends_with(b"\n") {
            end -= 1;
            if raw[..end].ends_with(b"\r") {
                end -= 1;
            }
        }
        // Validate every line so invalid UTF-8 anywhere in the file fails
        // exactly like the previous whole-file read_to_string did.
        let line = std::str::from_utf8(&raw[..end]).map_err(|_| {
            std::io::Error::new(
                std::io::ErrorKind::InvalidData,
                "stream did not contain valid UTF-8",
            )
        })?;
        if total_lines >= start_idx && window.len() < max_lines {
            window.push(line.to_string());
        }
        total_lines += 1;
    }

    Ok((window, total_lines))
}

/// Marker placed between the retained head and tail when a read window is
/// truncated by the byte budget. Mirrors qwen-code's truncation style so the
/// model sees both ends of the range.
const BYTE_TRUNCATION_SEPARATOR: &str = "\n\n---\n... [CONTENT TRUNCATED] ...\n---\n\n";

View on GitHub (pinned to 0c42157ee5)

Solutions

1. Confirm the encoding first: `file -i path` or `iconv -f UTF-8 -t UTF-8 path > /dev/null` to locate the bad bytes
2. Convert the file: `iconv -f WINDOWS-1252 -t UTF-8` (or the actual source encoding) and retry
3. If the file is intentionally binary, do not read it with this UTF-8 tool — use a byte-oriented path instead

Example fix

# before
$ file -i notes.txt
notes.txt: text/plain; charset=iso-8859-1   # read tool -> InvalidData

# after
$ iconv -f ISO-8859-1 -t UTF-8 notes.txt > notes.utf8.txt && mv notes.utf8.txt notes.txt

// Sniff UTF-8 before handing a path to the read tool.
let mut probe = vec![0u8; 8192];
let n = std::fs::File::open(&path)?.read(&mut probe)?;
if std::str::from_utf8(&probe[..n]).is_err() {
    // convert first (iconv) or refuse with an encoding hint
}

fn is_not_utf8_read(e: &std::io::Error) -> bool {
    e.kind() == std::io::ErrorKind::InvalidData && e.to_string().contains("valid UTF-8")
}

match read_lines(&path, start, max) {
    Ok(w) => Ok(w),
    Err(e) if is_not_utf8_read(&e) => Err(hint("file is not UTF-8; run `file -i` and iconv -t UTF-8")),
    Err(e) => Err(e),
}

When it happens

Trigger: Reading (or reading a line range of) a file that contains non-UTF-8 bytes anywhere: Latin-1/Windows-1252 text, GBK/Shift-JIS encoded source, or a binary file — the check fires even if the invalid byte is outside the requested start_idx window.

Common situations: Legacy source files saved in a pre-UTF-8 encoding; files generated by Windows tools; log files with mixed encodings; accidentally pointing the read tool at images/database dumps.

Related errors

• file is not UTF-8
• parallel(): expected an array of thunks
• private lane environment {} exceeds {} bytes
• {err}
• {err}

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/1fa23e614b4403d3. Report an issue: GitHub.