{"record":{"id":"c0d3c1f83dbd2528","repo":"uutils/coreutils","slug":"csplit-stream-not-utf8","errorCode":null,"errorMessage":"csplit-stream-not-utf8","messagePattern":"csplit-stream-not-utf8","errorType":"error_code","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"src/uu/csplit/src/csplit.rs","lineNumber":93,"sourceCode":"}\n\npub struct LinesWithNewlines<T: BufRead> {\n    inner: T,\n}\n\nimpl<T: BufRead> LinesWithNewlines<T> {\n    fn new(s: T) -> Self {\n        Self { inner: s }\n    }\n}\n\nimpl<T: BufRead> Iterator for LinesWithNewlines<T> {\n    type Item = io::Result<String>;\n\n    fn next(&mut self) -> Option<Self::Item> {\n        fn ret(v: Vec<u8>) -> io::Result<String> {\n            String::from_utf8(v).map_err(|_| {\n                io::Error::new(ErrorKind::InvalidData, translate!(\"csplit-stream-not-utf8\"))\n            })\n        }\n\n        let mut v = Vec::new();\n        match self.inner.read_until(b'\\n', &mut v) {\n            Ok(0) => None,\n            Ok(_) => Some(ret(v)),\n            Err(e) => Some(Err(e)),\n        }\n    }\n}\n\n/// Splits a file into severals according to the command line patterns.\n///\n/// # Errors\n///\n/// - [`io::Error`] if there is some problem reading/writing from/to a file.\n/// - [`CsplitError::LineOutOfRange`] if the line number pattern is larger than the number of input","sourceCodeStart":75,"sourceCodeEnd":111,"githubUrl":"https://github.com/uutils/coreutils/blob/92995feca29e324a7c559ede91db81c32d9f5d59/src/uu/csplit/src/csplit.rs#L75-L111","documentation":"`csplit` reads its input stream line-by-line as raw bytes via `read_until`, then converts each line to a Rust `String`. If the input is not valid UTF-8, `String::from_utf8` fails and this `InvalidData` error is produced, aborting the split.","triggerScenarios":"Feeding csplit a file (stdin, pipe, or path) that contains non-UTF-8 bytes, e.g. Latin-1 encoded text or binary data.","commonSituations":"Processing files created on Windows with legacy encodings (CP-1252), logs with embedded binary, or piping output of a tool that emits raw bytes.","solutions":["Convert the input file to UTF-8 first (e.g. `iconv -f latin1 -t utf8 input -o input.utf8`)","Use `csplit --encoding` if available, or a tool that supports byte-oriented splitting","Validate the input with `file` or `iconv -f utf8 -t utf8 input >/dev/null` before running","If the data is binary, use `split` instead of `csplit`"],"exampleFix":"// shell before\ncsplit pattern.txt '/regex/' '{*}'\n// after\niconv -f latin1 -t utf8 pattern.txt | csplit - '/regex/' '{*}'","handlingStrategy":"validation","validationCode":"iconv -f utf8 -t utf8 input.txt > /dev/null 2>&1 || echo 'input is not UTF-8'","typeGuard":"fn is_utf8(bytes: &[u8]) -> bool { std::str::from_utf8(bytes).is_ok() }","tryCatchPattern":"match line_result {\n    Err(e) if e.kind() == ErrorKind::InvalidData => {\n        eprintln!(\"csplit input must be UTF-8: {e}\");\n        std::process::exit(1);\n    }\n    r => r?,\n}","preventionTips":["Convert inputs to UTF-8 with iconv/dos2unix before splitting","Check file encoding with `file -bi` in scripts","Avoid piping binary data into csplit","Add a pre-flight UTF-8 validation step in pipelines"],"tags":["encoding","utf8","csplit"],"backgroundTag":"invalid-argument-format","analyzedSha":"92995feca29e324a7c559ede91db81c32d9f5d59","analyzedAt":"2026-09-20T12:39:35.476Z","contentChangedAt":"2026-09-20T12:39:35.476Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}