{"record":{"id":"1fa23e614b4403d3","repo":"Hmbown/CodeWhale","slug":"stream-did-not-contain-valid-utf-8","errorCode":null,"errorMessage":"stream did not contain valid UTF-8","messagePattern":"stream did not contain valid UTF-8","errorType":"validation","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"crates/tui/src/tools/file.rs","lineNumber":1210,"sourceCode":"    loop {\n        raw.clear();\n        let n = reader.read_until(b'\\n', &mut raw)?;\n        if n == 0 {\n            break;\n        }\n        // Mirror `str::lines`: strip the trailing '\\n', and a '\\r' only when\n        // it directly precedes that '\\n'.\n        let mut end = raw.len();\n        if raw[..end].ends_with(b\"\\n\") {\n            end -= 1;\n            if raw[..end].ends_with(b\"\\r\") {\n                end -= 1;\n            }\n        }\n        // Validate every line so invalid UTF-8 anywhere in the file fails\n        // exactly like the previous whole-file read_to_string did.\n        let line = std::str::from_utf8(&raw[..end]).map_err(|_| {\n            std::io::Error::new(\n                std::io::ErrorKind::InvalidData,\n                \"stream did not contain valid UTF-8\",\n            )\n        })?;\n        if total_lines >= start_idx && window.len() < max_lines {\n            window.push(line.to_string());\n        }\n        total_lines += 1;\n    }\n\n    Ok((window, total_lines))\n}\n\n/// Marker placed between the retained head and tail when a read window is\n/// truncated by the byte budget. Mirrors qwen-code's truncation style so the\n/// model sees both ends of the range.\nconst BYTE_TRUNCATION_SEPARATOR: &str = \"\\n\\n---\\n... [CONTENT TRUNCATED] ...\\n---\\n\\n\";\n","sourceCodeStart":1192,"sourceCodeEnd":1228,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/src/tools/file.rs#L1192-L1228","documentation":"read_window_streaming validates each decoded line with std::str::from_utf8 so that invalid UTF-8 anywhere in the file fails exactly like the previous whole-file read_to_string did. When a line's bytes are not valid UTF-8 it raises io::ErrorKind::InvalidData with this message and aborts the read. Binary files (images, compiled artifacts) are therefore rejected by the file read tool by design.","triggerScenarios":"Calling the file-read tool's execute -> read_window_streaming on a file whose bytes in the requested window (or any earlier line) are not valid UTF-8 — e.g. images, PDFs, executables, or text in a non-UTF-8 encoding like Latin-1.","commonSituations":"Pointing the read tool at a PNG/JPEG/binary log; reading legacy text files saved in ISO-8859-1 or Windows-1252; files with a BOM-mangled or mixed encoding.","solutions":["Confirm the target file is actually text (`file <path>`); if it is binary, read metadata instead of content.","Convert the file to UTF-8 first: `iconv -f LATIN1 -t UTF-8 input.txt > input.utf8.txt`.","If only part of the file is bad, re-encode or strip it, or read a window that excludes the invalid region after conversion.","If the tool must tolerate it, handle the InvalidData error and fall back to a binary-safe preview (hex dump or lossy decode)."],"exampleFix":"// before\nread(\"data/export.csv\")  // export.csv saved as Windows-1252\n// after\n// terminal\niconv -f WINDOWS-1252 -t UTF-8 data/export.csv > data/export.utf8.csv\n// then read(\"data/export.utf8.csv\")","handlingStrategy":"validation","validationCode":"// before passing a path to the read tool, ensure it is text:\n// shell\nfile --mime-encoding <path>   # expect us-ascii / utf-8, not binary/iso-8859-1","typeGuard":"fn is_valid_utf8_file(bytes: &[u8]) -> bool { std::str::from_utf8(bytes).is_ok() }","tryCatchPattern":"// rust\nmatch read_window_streaming(...) {\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData => {\n        eprintln!(\"file is not valid UTF-8; convert with iconv or read as binary\");\n    }\n    other => other?,\n}","preventionTips":["Run `file` or a UTF-8 probe on unfamiliar files before reading them with the tool.","Convert non-UTF-8 text with iconv or dos2unix before use.","Never point a text-only reader at binaries (images, PDFs, executables); use metadata tools instead."],"tags":["encoding","utf-8","file-read","io"],"backgroundTag":"invalid-utf8-data","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}