Hmbown/CodeWhale · error · io::Error

stream did not contain valid UTF-8

Error message

stream did not contain valid UTF-8

What it means

read_window_streaming validates each decoded line with std::str::from_utf8 so that invalid UTF-8 anywhere in the file fails exactly like the previous whole-file read_to_string did. When a line's bytes are not valid UTF-8 it raises io::ErrorKind::InvalidData with this message and aborts the read. Binary files (images, compiled artifacts) are therefore rejected by the file read tool by design.

Solutions

1. Confirm the target file is actually text (`file <path>`); if it is binary, read metadata instead of content.
2. Convert the file to UTF-8 first: `iconv -f LATIN1 -t UTF-8 input.txt > input.utf8.txt`.
3. If only part of the file is bad, re-encode or strip it, or read a window that excludes the invalid region after conversion.
4. If the tool must tolerate it, handle the InvalidData error and fall back to a binary-safe preview (hex dump or lossy decode).

Example fix

// before
read("data/export.csv")  // export.csv saved as Windows-1252
// after
// terminal
iconv -f WINDOWS-1252 -t UTF-8 data/export.csv > data/export.utf8.csv
// then read("data/export.utf8.csv")

// before passing a path to the read tool, ensure it is text:
// shell
file --mime-encoding <path>   # expect us-ascii / utf-8, not binary/iso-8859-1

fn is_valid_utf8_file(bytes: &[u8]) -> bool { std::str::from_utf8(bytes).is_ok() }

// rust
match read_window_streaming(...) {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {
        eprintln!("file is not valid UTF-8; convert with iconv or read as binary");
    }
    other => other?,
}

When it happens

Trigger: Calling the file-read tool's execute -> read_window_streaming on a file whose bytes in the requested window (or any earlier line) are not valid UTF-8 — e.g. images, PDFs, executables, or text in a non-UTF-8 encoding like Latin-1.

Common situations: Pointing the read tool at a PNG/JPEG/binary log; reading legacy text files saved in ISO-8859-1 or Windows-1252; files with a BOM-mangled or mixed encoding.

Related errors

• file is not UTF-8
• file appears to be binary
• artifact name is not portable UTF-8
• child stdout was not valid UTF-8
• Codewhale-owned xAI OAuth file

AI-assisted analysis of Hmbown/CodeWhale@73e0f67d83 (2026-09-22). Data as JSON: /api/errors/1fa23e614b4403d3. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/src/tools/file.rs:1210

    loop {
        raw.clear();
        let n = reader.read_until(b'\n', &mut raw)?;
        if n == 0 {
            break;
        }
        // Mirror `str::lines`: strip the trailing '\n', and a '\r' only when
        // it directly precedes that '\n'.
        let mut end = raw.len();
        if raw[..end].ends_with(b"\n") {
            end -= 1;
            if raw[..end].ends_with(b"\r") {
                end -= 1;
            }
        }
        // Validate every line so invalid UTF-8 anywhere in the file fails
        // exactly like the previous whole-file read_to_string did.
        let line = std::str::from_utf8(&raw[..end]).map_err(|_| {
            std::io::Error::new(
                std::io::ErrorKind::InvalidData,
                "stream did not contain valid UTF-8",
            )
        })?;
        if total_lines >= start_idx && window.len() < max_lines {
            window.push(line.to_string());
        }
        total_lines += 1;
    }

    Ok((window, total_lines))
}

/// Marker placed between the retained head and tail when a read window is
/// truncated by the byte budget. Mirrors qwen-code's truncation style so the
/// model sees both ends of the range.
const BYTE_TRUNCATION_SEPARATOR: &str = "\n\n---\n... [CONTENT TRUNCATED] ...\n---\n\n";

View on GitHub (pinned to 73e0f67d83)