{"record":{"id":"492b567a2cff4161","repo":"Hmbown/CodeWhale","slug":"file-is-not-utf-8","errorCode":null,"errorMessage":"file is not UTF-8","messagePattern":"file is not UTF-8","errorType":"validation","errorClass":"io::Error","httpStatus":null,"severity":"warning","filePath":"crates/tui/src/tui/file_mention.rs","lineNumber":1419,"sourceCode":"        // multi-byte sequence (CJK, emoji, etc.) is never split at the cut point.\n        // Only adjust when error_len() is None — that means truncation landed\n        // mid-sequence (incomplete tail).  A Some(_) error_len means the file\n        // genuinely contains invalid UTF-8 bytes; leave the buffer intact so\n        // the from_utf8 call below returns the correct \"file is not UTF-8\" error.\n        if let Err(e) = std::str::from_utf8(&buffer)\n            && e.error_len().is_none()\n        {\n            buffer.truncate(e.valid_up_to());\n        }\n    }\n    if buffer.contains(&0) {\n        return Err(std::io::Error::new(\n            std::io::ErrorKind::InvalidData,\n            \"file appears to be binary\",\n        ));\n    }\n    let text = std::str::from_utf8(&buffer)\n        .map_err(|_| std::io::Error::new(std::io::ErrorKind::InvalidData, \"file is not UTF-8\"))?\n        .to_string();\n    Ok((text, truncated))\n}\n\nfn is_media_path(path: &Path) -> bool {\n    let Some(ext) = path.extension().and_then(|ext| ext.to_str()) else {\n        return false;\n    };\n    matches!(\n        ext.to_ascii_lowercase().as_str(),\n        \"png\"\n            | \"jpg\"\n            | \"jpeg\"\n            | \"gif\"\n            | \"webp\"\n            | \"bmp\"\n            | \"tif\"\n            | \"tiff\"","sourceCodeStart":1401,"sourceCodeEnd":1437,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/src/tui/file_mention.rs#L1401-L1437","documentation":"After the NUL-byte binary check passes, read_text_prefix converts the buffer with str::from_utf8 and maps failure to InvalidData 'file is not UTF-8'. The file looks textual but contains bytes that are not valid UTF-8 (e.g. Latin-1, UTF-16, or a truncated multibyte sequence at the read boundary).","triggerScenarios":"Reading a file encoded in a non-UTF-8 charset (Windows-1252, ISO-8859-1, UTF-16) or one whose read buffer splits a multibyte UTF-8 sequence in a way that leaves invalid trailing bytes.","commonSituations":"Files created on Windows with ANSI encodings; data exported from legacy tools; logs mixing encodings; files edited in tools that rewrote encoding.","solutions":["Re-encode the file to UTF-8 (e.g. `iconv -f latin1 -t utf8 file -o file.utf8`)","Use a lossy reader (String::from_utf8_lossy) at the call site if approximate content is acceptable","Detect encoding before read and transcode accordingly","Handle the InvalidData error and inform the user the file is not UTF-8 text"],"exampleFix":"// before\nlet (text, truncated) = read_text_prefix(&path, limit)?;\n// after\nlet bytes = read_bytes_prefix(&path, limit)?;\nlet text = String::from_utf8_lossy(&bytes).into_owned(); // lossy instead of erroring","handlingStrategy":"fallback","validationCode":"let bytes = std::fs::read(&path)?;\nif std::str::from_utf8(&bytes).is_err() {\n    // re-encode or use a lossy path instead of read_text_prefix\n}","typeGuard":"fn is_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}","tryCatchPattern":"match read_text_prefix(&path, limit) {\n    Err(e) if e.kind() == io::ErrorKind::InvalidData => String::from_utf8_lossy(&bytes).into_owned(),\n    Ok((text, trunc)) => text,\n    Err(e) => return Err(e),\n}","preventionTips":["Standardize on UTF-8 for all project files","Re-encode legacy files with iconv during import","Use lossy conversion where approximate content is acceptable","Never concatenate files of mixed encodings"],"tags":["io","encoding","utf-8","file-read"],"backgroundTag":"file-read-failed","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}