uutils/coreutils · error · io::Error

csplit-stream-not-utf8

Error message

csplit-stream-not-utf8

What it means

csplit reads its input as lines via LinesWithNewlines, which requires valid UTF-8. When read_until collects bytes that are not valid UTF-8, String::from_utf8 fails and this InvalidData io::Error is produced. GNU csplit is byte-oriented, but this implementation works on Strings, so binary/non-UTF-8 input is unsupported.

Source

Thrown at src/uu/csplit/src/csplit.rs:91

}

pub struct LinesWithNewlines<T: BufRead> {
    inner: T,
}

impl<T: BufRead> LinesWithNewlines<T> {
    fn new(s: T) -> Self {
        Self { inner: s }
    }
}

impl<T: BufRead> Iterator for LinesWithNewlines<T> {
    type Item = io::Result<String>;

    fn next(&mut self) -> Option<Self::Item> {
        fn ret(v: Vec<u8>) -> io::Result<String> {
            String::from_utf8(v).map_err(|_| {
                io::Error::new(ErrorKind::InvalidData, translate!("csplit-stream-not-utf8"))
            })
        }

        let mut v = Vec::new();
        match self.inner.read_until(b'\n', &mut v) {
            Ok(0) => None,
            Ok(_) => Some(ret(v)),
            Err(e) => Some(Err(e)),
        }
    }
}

/// Splits a file into severals according to the command line patterns.
///
/// # Errors
///
/// - [`io::Error`] if there is some problem reading/writing from/to a file.
/// - [`CsplitError::LineOutOfRange`] if the line number pattern is larger than the number of input

View on GitHub (pinned to 9ff4114e82)

Solutions

1. Convert the input to UTF-8 first: iconv -f ISO-8859-1 -t UTF-8 input > input.utf8
2. Identify the offending bytes with `grep -naP '[\x80-\xFF]' file` or `file`/`chardet` and fix the source encoding
3. If the data is binary, use a byte-oriented splitter (e.g., csplit from GNU coreutils or split) instead
4. Re-generate the input with an explicit UTF-8 encoding

Example fix

// before
csplit input.log '/ERROR/'
// after
iconv -f WINDOWS-1252 -t UTF-8 input.log > input-utf8.log
csplit input-utf8.log '/ERROR/'

// validate input encoding before running csplit
if let Err(bad) = std::str::from_utf8(&std::fs::read("input.log")?) {
    eprintln!("input is not UTF-8 at byte offset {}", bad.valid_up_to());
    // convert with iconv first
}

fn is_utf8(bytes: &[u8]) -> bool { std::str::from_utf8(bytes).is_ok() }

null

When it happens

Trigger: Running csplit on a file containing non-UTF-8 bytes (e.g., Latin-1 encoded text, binary data, or a file with a BOM/encoding other than UTF-8).

Common situations: Splitting logs or data files produced on Windows in CP-1252/Latin-1, splitting files downloaded in a legacy encoding, accidentally csplit-ing a binary file.

Related errors

• InvalidUtf8 clap error for non-UTF-8 argument value
• csplit-write-split-not-created

AI-assisted analysis of uutils/coreutils@9ff4114e82 (2026-08-31). Data as JSON: /api/errors/c0d3c1f83dbd2528. Report an issue: GitHub.