{"record":{"id":"1fa23e614b4403d3","repo":"Hmbown/CodeWhale","slug":"stream-did-not-contain-valid-utf-8","errorCode":null,"errorMessage":"stream did not contain valid UTF-8","messagePattern":"stream did not contain valid UTF-8","errorType":"exception","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"crates/tui/src/tools/file.rs","lineNumber":1042,"sourceCode":"    loop {\n        raw.clear();\n        let n = reader.read_until(b'\\n', &mut raw)?;\n        if n == 0 {\n            break;\n        }\n        // Mirror `str::lines`: strip the trailing '\\n', and a '\\r' only when\n        // it directly precedes that '\\n'.\n        let mut end = raw.len();\n        if raw[..end].ends_with(b\"\\n\") {\n            end -= 1;\n            if raw[..end].ends_with(b\"\\r\") {\n                end -= 1;\n            }\n        }\n        // Validate every line so invalid UTF-8 anywhere in the file fails\n        // exactly like the previous whole-file read_to_string did.\n        let line = std::str::from_utf8(&raw[..end]).map_err(|_| {\n            std::io::Error::new(\n                std::io::ErrorKind::InvalidData,\n                \"stream did not contain valid UTF-8\",\n            )\n        })?;\n        if total_lines >= start_idx && window.len() < max_lines {\n            window.push(line.to_string());\n        }\n        total_lines += 1;\n    }\n\n    Ok((window, total_lines))\n}\n\n/// Marker placed between the retained head and tail when a read window is\n/// truncated by the byte budget. Mirrors qwen-code's truncation style so the\n/// model sees both ends of the range.\nconst BYTE_TRUNCATION_SEPARATOR: &str = \"\\n\\n---\\n... [CONTENT TRUNCATED] ...\\n---\\n\\n\";\n","sourceCodeStart":1024,"sourceCodeEnd":1060,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/0c42157ee52f9d55af2b506d71b46249910f77d3/crates/tui/src/tools/file.rs#L1024-L1060","documentation":"InvalidData with std's classic message, raised by the file-reading tool in tools/file.rs when any line read from the file is not valid UTF-8. The windowed reader validates every line exactly so behavior matches the old whole-file read_to_string: one invalid byte anywhere in the file fails the read, not just in the requested window.","triggerScenarios":"Reading (or reading a line range of) a file that contains non-UTF-8 bytes anywhere: Latin-1/Windows-1252 text, GBK/Shift-JIS encoded source, or a binary file — the check fires even if the invalid byte is outside the requested start_idx window.","commonSituations":"Legacy source files saved in a pre-UTF-8 encoding; files generated by Windows tools; log files with mixed encodings; accidentally pointing the read tool at images/database dumps.","solutions":["Confirm the encoding first: `file -i path` or `iconv -f UTF-8 -t UTF-8 path > /dev/null` to locate the bad bytes","Convert the file: `iconv -f WINDOWS-1252 -t UTF-8` (or the actual source encoding) and retry","If the file is intentionally binary, do not read it with this UTF-8 tool — use a byte-oriented path instead"],"exampleFix":"# before\n$ file -i notes.txt\nnotes.txt: text/plain; charset=iso-8859-1   # read tool -> InvalidData\n\n# after\n$ iconv -f ISO-8859-1 -t UTF-8 notes.txt > notes.utf8.txt && mv notes.utf8.txt notes.txt","handlingStrategy":"validation","validationCode":"// Sniff UTF-8 before handing a path to the read tool.\nlet mut probe = vec![0u8; 8192];\nlet n = std::fs::File::open(&path)?.read(&mut probe)?;\nif std::str::from_utf8(&probe[..n]).is_err() {\n    // convert first (iconv) or refuse with an encoding hint\n}","typeGuard":"fn is_not_utf8_read(e: &std::io::Error) -> bool {\n    e.kind() == std::io::ErrorKind::InvalidData && e.to_string().contains(\"valid UTF-8\")\n}","tryCatchPattern":"match read_lines(&path, start, max) {\n    Ok(w) => Ok(w),\n    Err(e) if is_not_utf8_read(&e) => Err(hint(\"file is not UTF-8; run `file -i` and iconv -t UTF-8\")),\n    Err(e) => Err(e),\n}","preventionTips":["Run `file -i` on unfamiliar files before reading them","Convert legacy-encoding files to UTF-8 at import time, not read time","Remember the check is whole-file: a bad byte outside your requested line window still fails the read"],"tags":["file-io","encoding","utf-8","rust","read-tool"],"backgroundTag":"invalid-utf-8-file","analyzedSha":"0c42157ee52f9d55af2b506d71b46249910f77d3","analyzedAt":"2026-08-20T21:50:45.477Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}