{"record":{"id":"492b567a2cff4161","repo":"Hmbown/CodeWhale","slug":"file-is-not-utf-8","errorCode":null,"errorMessage":"file is not UTF-8","messagePattern":"file is not UTF-8","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"warning","filePath":"crates/tui/src/tui/file_mention.rs","lineNumber":1406,"sourceCode":"        // multi-byte sequence (CJK, emoji, etc.) is never split at the cut point.\n        // Only adjust when error_len() is None — that means truncation landed\n        // mid-sequence (incomplete tail).  A Some(_) error_len means the file\n        // genuinely contains invalid UTF-8 bytes; leave the buffer intact so\n        // the from_utf8 call below returns the correct \"file is not UTF-8\" error.\n        if let Err(e) = std::str::from_utf8(&buffer)\n            && e.error_len().is_none()\n        {\n            buffer.truncate(e.valid_up_to());\n        }\n    }\n    if buffer.contains(&0) {\n        return Err(std::io::Error::new(\n            std::io::ErrorKind::InvalidData,\n            \"file appears to be binary\",\n        ));\n    }\n    let text = std::str::from_utf8(&buffer)\n        .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, \"file is not UTF-8\"))?\n        .to_string();\n    Ok((text, truncated))\n}\n\nfn is_media_path(path: &Path) -> bool {\n    let Some(ext) = path.extension().and_then(|ext| ext.to_str()) else {\n        return false;\n    };\n    matches!(\n        ext.to_ascii_lowercase().as_str(),\n        \"png\"\n            | \"jpg\"\n            | \"jpeg\"\n            | \"gif\"\n            | \"webp\"\n            | \"bmp\"\n            | \"tif\"\n            | \"tiff\"","sourceCodeStart":1388,"sourceCodeEnd":1424,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/0c42157ee52f9d55af2b506d71b46249910f77d3/crates/tui/src/tui/file_mention.rs#L1388-L1424","documentation":"InvalidData ('file is not UTF-8') raised by the @-mention reader when the buffer has no NUL byte but still fails str::from_utf8. This is the text-file sibling of the binary check: the earlier truncation fix-up only repairs a cut that lands mid-multibyte-sequence, so reaching this error means the file genuinely contains invalid UTF-8 bytes (a Some(_) error_len), such as a legacy 8-bit encoding.","triggerScenarios":"@-mentioning a text file saved in Latin-1/Windows-1252/GBK/Shift-JIS (no NUL bytes, but byte sequences that are not valid UTF-8).","commonSituations":"Old source files or logs from Windows-era tools; files from colleagues on different locale systems; mixed-encoding concatenations; mojibake files where broken bytes were saved back.","solutions":["Identify the real encoding: `file -i path`","Convert to UTF-8: `iconv -f WINDOWS-1252 -t UTF-8 in > out` (substitute the detected encoding), then mention the converted file","For one-off inspection, keep the original untouched and mention a converted copy"],"exampleFix":"# before\n$ file -i readme.txt\nreadme.txt: text/plain; charset=iso-8859-1   # @-mention -> InvalidData: file is not UTF-8\n\n# after\n$ iconv -f ISO-8859-1 -t UTF-8 readme.txt > readme.utf8.txt\n# @-mention readme.utf8.txt","handlingStrategy":"validation","validationCode":"let bytes = std::fs::read(p)?;\nmatch std::str::from_utf8(&bytes) {\n    Ok(_) => { /* safe to @-mention */ }\n    Err(e) if e.error_len().is_none() => { /* truncated tail only; retry after full read */ }\n    Err(_) => { /* convert encoding first (iconv), then mention */ }\n}","typeGuard":"fn is_not_utf8_mention(e: &std::io::Error) -> bool {\n    e.kind() == std::io::ErrorKind::InvalidData && e.to_string().contains(\"file is not UTF-8\")\n}","tryCatchPattern":null,"preventionTips":["Standardize repos on UTF-8 and enforce with editorconfig/pre-commit","Convert Windows-locale files at import time","Check `file -i` when a mention unexpectedly fails"],"tags":["file-mention","encoding","utf-8","tui","rust"],"backgroundTag":"invalid-utf-8-file","analyzedSha":"0c42157ee52f9d55af2b506d71b46249910f77d3","analyzedAt":"2026-08-20T21:50:45.477Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}