Hmbown/CodeWhale · warning · io::Error

file is not UTF-8

Error message

file is not UTF-8

What it means

After the NUL-byte binary check passes, read_text_prefix converts the buffer with str::from_utf8 and maps failure to InvalidData 'file is not UTF-8'. The file looks textual but contains bytes that are not valid UTF-8 (e.g. Latin-1, UTF-16, or a truncated multibyte sequence at the read boundary).

Solutions

1. Re-encode the file to UTF-8 (e.g. `iconv -f latin1 -t utf8 file -o file.utf8`)
2. Use a lossy reader (String::from_utf8_lossy) at the call site if approximate content is acceptable
3. Detect encoding before read and transcode accordingly
4. Handle the InvalidData error and inform the user the file is not UTF-8 text

Example fix

// before
let (text, truncated) = read_text_prefix(&path, limit)?;
// after
let bytes = read_bytes_prefix(&path, limit)?;
let text = String::from_utf8_lossy(&bytes).into_owned(); // lossy instead of erroring

let bytes = std::fs::read(&path)?;
if std::str::from_utf8(&bytes).is_err() {
    // re-encode or use a lossy path instead of read_text_prefix
}

fn is_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}

match read_text_prefix(&path, limit) {
    Err(e) if e.kind() == io::ErrorKind::InvalidData => String::from_utf8_lossy(&bytes).into_owned(),
    Ok((text, trunc)) => text,
    Err(e) => return Err(e),
}

When it happens

Trigger: Reading a file encoded in a non-UTF-8 charset (Windows-1252, ISO-8859-1, UTF-16) or one whose read buffer splits a multibyte UTF-8 sequence in a way that leaves invalid trailing bytes.

Common situations: Files created on Windows with ANSI encodings; data exported from legacy tools; logs mixing encodings; files edited in tools that rewrote encoding.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• stream did not contain valid UTF-8
• file appears to be binary
• artifact name is not portable UTF-8
• child stdout was not valid UTF-8
• Codewhale-owned xAI OAuth file

AI-assisted analysis of Hmbown/CodeWhale@73e0f67d83 (2026-09-22). Data as JSON: /api/errors/492b567a2cff4161. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/src/tui/file_mention.rs:1419

        // multi-byte sequence (CJK, emoji, etc.) is never split at the cut point.
        // Only adjust when error_len() is None — that means truncation landed
        // mid-sequence (incomplete tail).  A Some(_) error_len means the file
        // genuinely contains invalid UTF-8 bytes; leave the buffer intact so
        // the from_utf8 call below returns the correct "file is not UTF-8" error.
        if let Err(e) = std::str::from_utf8(&buffer)
            && e.error_len().is_none()
        {
            buffer.truncate(e.valid_up_to());
        }
    }
    if buffer.contains(&0) {
        return Err(std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            "file appears to be binary",
        ));
    }
    let text = std::str::from_utf8(&buffer)
        .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, "file is not UTF-8"))?
        .to_string();
    Ok((text, truncated))
}

fn is_media_path(path: &Path) -> bool {
    let Some(ext) = path.extension().and_then(|ext| ext.to_str()) else {
        return false;
    };
    matches!(
        ext.to_ascii_lowercase().as_str(),
        "png"
            | "jpg"
            | "jpeg"
            | "gif"
            | "webp"
            | "bmp"
            | "tif"
            | "tiff"

View on GitHub (pinned to 73e0f67d83)