Hmbown/CodeWhale · warning · std::io::Error

file is not UTF-8

Error message

file is not UTF-8

What it means

InvalidData ('file is not UTF-8') raised by the @-mention reader when the buffer has no NUL byte but still fails str::from_utf8. This is the text-file sibling of the binary check: the earlier truncation fix-up only repairs a cut that lands mid-multibyte-sequence, so reaching this error means the file genuinely contains invalid UTF-8 bytes (a Some(_) error_len), such as a legacy 8-bit encoding.

Source

Thrown at crates/tui/src/tui/file_mention.rs:1406

        // multi-byte sequence (CJK, emoji, etc.) is never split at the cut point.
        // Only adjust when error_len() is None — that means truncation landed
        // mid-sequence (incomplete tail).  A Some(_) error_len means the file
        // genuinely contains invalid UTF-8 bytes; leave the buffer intact so
        // the from_utf8 call below returns the correct "file is not UTF-8" error.
        if let Err(e) = std::str::from_utf8(&buffer)
            && e.error_len().is_none()
        {
            buffer.truncate(e.valid_up_to());
        }
    }
    if buffer.contains(&0) {
        return Err(std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            "file appears to be binary",
        ));
    }
    let text = std::str::from_utf8(&buffer)
        .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, "file is not UTF-8"))?
        .to_string();
    Ok((text, truncated))
}

fn is_media_path(path: &Path) -> bool {
    let Some(ext) = path.extension().and_then(|ext| ext.to_str()) else {
        return false;
    };
    matches!(
        ext.to_ascii_lowercase().as_str(),
        "png"
            | "jpg"
            | "jpeg"
            | "gif"
            | "webp"
            | "bmp"
            | "tif"
            | "tiff"

View on GitHub (pinned to 0c42157ee5)

Solutions

1. Identify the real encoding: `file -i path`
2. Convert to UTF-8: `iconv -f WINDOWS-1252 -t UTF-8 in > out` (substitute the detected encoding), then mention the converted file
3. For one-off inspection, keep the original untouched and mention a converted copy

Example fix

# before
$ file -i readme.txt
readme.txt: text/plain; charset=iso-8859-1   # @-mention -> InvalidData: file is not UTF-8

# after
$ iconv -f ISO-8859-1 -t UTF-8 readme.txt > readme.utf8.txt
# @-mention readme.utf8.txt

let bytes = std::fs::read(p)?;
match std::str::from_utf8(&bytes) {
    Ok(_) => { /* safe to @-mention */ }
    Err(e) if e.error_len().is_none() => { /* truncated tail only; retry after full read */ }
    Err(_) => { /* convert encoding first (iconv), then mention */ }
}

fn is_not_utf8_mention(e: &std::io::Error) -> bool {
    e.kind() == std::io::ErrorKind::InvalidData && e.to_string().contains("file is not UTF-8")
}

When it happens

Trigger: @-mentioning a text file saved in Latin-1/Windows-1252/GBK/Shift-JIS (no NUL bytes, but byte sequences that are not valid UTF-8).

Common situations: Old source files or logs from Windows-era tools; files from colleagues on different locale systems; mixed-encoding concatenations; mojibake files where broken bytes were saved back.

Related errors

• stream did not contain valid UTF-8
• file appears to be binary
• unsupported locale '{other}'
• invalid locale '{value}'
• unsupported theme '{other}'

AI-assisted analysis of Hmbown/CodeWhale@0c42157ee5 (2026-08-20). Data as JSON: /api/errors/492b567a2cff4161. Report an issue: GitHub.