{"record":{"id":"c0d3c1f83dbd2528","repo":"uutils/coreutils","slug":"csplit-stream-not-utf8","errorCode":null,"errorMessage":"csplit-stream-not-utf8","messagePattern":"csplit-stream-not-utf8","errorType":"error_code","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"src/uu/csplit/src/csplit.rs","lineNumber":91,"sourceCode":"}\n\npub struct LinesWithNewlines<T: BufRead> {\n    inner: T,\n}\n\nimpl<T: BufRead> LinesWithNewlines<T> {\n    fn new(s: T) -> Self {\n        Self { inner: s }\n    }\n}\n\nimpl<T: BufRead> Iterator for LinesWithNewlines<T> {\n    type Item = io::Result<String>;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        fn ret(v: Vec<u8>) -> io::Result<String> {\n            String::from_utf8(v).map_err(|_| {\n                io::Error::new(ErrorKind::InvalidData, translate!(\"csplit-stream-not-utf8\"))\n            })\n        }\n\n        let mut v = Vec::new();\n        match self.inner.read_until(b'\\n', &mut v) {\n            Ok(0) => None,\n            Ok(_) => Some(ret(v)),\n            Err(e) => Some(Err(e)),\n        }\n    }\n}\n\n/// Splits a file into severals according to the command line patterns.\n///\n/// # Errors\n///\n/// - [`io::Error`] if there is some problem reading/writing from/to a file.\n/// - [`CsplitError::LineOutOfRange`] if the line number pattern is larger than the number of input","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/uutils/coreutils/blob/9ff4114e82c8fb80420fd4fb6a319bd3227095e0/src/uu/csplit/src/csplit.rs#L73-L109","documentation":"csplit reads its input as lines via LinesWithNewlines, which requires valid UTF-8. When read_until collects bytes that are not valid UTF-8, String::from_utf8 fails and this InvalidData io::Error is produced. GNU csplit is byte-oriented, but this implementation works on Strings, so binary/non-UTF-8 input is unsupported.","triggerScenarios":"Running csplit on a file containing non-UTF-8 bytes (e.g., Latin-1 encoded text, binary data, or a file with a BOM/encoding other than UTF-8).","commonSituations":"Splitting logs or data files produced on Windows in CP-1252/Latin-1, splitting files downloaded in a legacy encoding, accidentally csplit-ing a binary file.","solutions":["Convert the input to UTF-8 first: iconv -f ISO-8859-1 -t UTF-8 input > input.utf8","Identify the offending bytes with `grep -naP '[\\x80-\\xFF]' file` or `file`/`chardet` and fix the source encoding","If the data is binary, use a byte-oriented splitter (e.g., csplit from GNU coreutils or split) instead","Re-generate the input with an explicit UTF-8 encoding"],"exampleFix":"// before\ncsplit input.log '/ERROR/'\n// after\niconv -f WINDOWS-1252 -t UTF-8 input.log > input-utf8.log\ncsplit input-utf8.log '/ERROR/'","handlingStrategy":"validation","validationCode":"// validate input encoding before running csplit\nif let Err(bad) = std::str::from_utf8(&std::fs::read(\"input.log\")?) {\n    eprintln!(\"input is not UTF-8 at byte offset {}\", bad.valid_up_to());\n    // convert with iconv first\n}\n","typeGuard":"fn is_utf8(bytes: &[u8]) -> bool { std::str::from_utf8(bytes).is_ok() }","tryCatchPattern":"null","preventionTips":["Ensure all input files are UTF-8 (enforce in editors/CI with a encoding check)","Never feed binary files to csplit; use split for byte-oriented splitting","Convert legacy encodings with iconv as an ingestion step"],"tags":["csplit","encoding","utf-8","invalid-data"],"backgroundTag":"invalid-utf8-input","analyzedSha":"9ff4114e82c8fb80420fd4fb6a319bd3227095e0","analyzedAt":"2026-08-31T11:11:36.175Z","contentChangedAt":"2026-08-31T11:11:36.175Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}