uutils/coreutils · error · std::io::Error

csplit-stream-not-utf8

Error message

csplit-stream-not-utf8

What it means

`csplit` reads its input stream line-by-line as raw bytes via `read_until`, then converts each line to a Rust `String`. If the input is not valid UTF-8, `String::from_utf8` fails and this `InvalidData` error is produced, aborting the split.

Solutions

1. Convert the input file to UTF-8 first (e.g. `iconv -f latin1 -t utf8 input -o input.utf8`)
2. Use `csplit --encoding` if available, or a tool that supports byte-oriented splitting
3. Validate the input with `file` or `iconv -f utf8 -t utf8 input >/dev/null` before running
4. If the data is binary, use `split` instead of `csplit`

Example fix

// shell before
csplit pattern.txt '/regex/' '{*}'
// after
iconv -f latin1 -t utf8 pattern.txt | csplit - '/regex/' '{*}'

iconv -f utf8 -t utf8 input.txt > /dev/null 2>&1 || echo 'input is not UTF-8'

fn is_utf8(bytes: &[u8]) -> bool { std::str::from_utf8(bytes).is_ok() }

match line_result {
    Err(e) if e.kind() == ErrorKind::InvalidData => {
        eprintln!("csplit input must be UTF-8: {e}");
        std::process::exit(1);
    }
    r => r?,
}

When it happens

Trigger: Feeding csplit a file (stdin, pipe, or path) that contains non-UTF-8 bytes, e.g. Latin-1 encoded text or binary data.

Common situations: Processing files created on Windows with legacy encodings (CP-1252), logs with embedded binary, or piping output of a tool that emits raw bytes.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• csplit-write-split-not-created
• InvalidUtf8

AI-assisted analysis of uutils/coreutils@92995feca2 (2026-09-20). Data as JSON: /api/errors/c0d3c1f83dbd2528. Report an issue: GitHub.

Appendix: source

Thrown at src/uu/csplit/src/csplit.rs:93

}

pub struct LinesWithNewlines<T: BufRead> {
    inner: T,
}

impl<T: BufRead> LinesWithNewlines<T> {
    fn new(s: T) -> Self {
        Self { inner: s }
    }
}

impl<T: BufRead> Iterator for LinesWithNewlines<T> {
    type Item = io::Result<String>;

    fn next(&mut self) -> Option<Self::Item> {
        fn ret(v: Vec<u8>) -> io::Result<String> {
            String::from_utf8(v).map_err(|_| {
                io::Error::new(ErrorKind::InvalidData, translate!("csplit-stream-not-utf8"))
            })
        }

        let mut v = Vec::new();
        match self.inner.read_until(b'\n', &mut v) {
            Ok(0) => None,
            Ok(_) => Some(ret(v)),
            Err(e) => Some(Err(e)),
        }
    }
}

/// Splits a file into severals according to the command line patterns.
///
/// # Errors
///
/// - [`io::Error`] if there is some problem reading/writing from/to a file.
/// - [`CsplitError::LineOutOfRange`] if the line number pattern is larger than the number of input

View on GitHub (pinned to 92995feca2)