can1357/oh-my-pi · error · io::Error

invalid utf-8 sequence

Error message

invalid utf-8 sequence

What it means

sed's `OutputBuffer` implements std::io::Write by converting every written byte slice with `std::str::from_utf8`; if the bytes are not valid UTF-8 it returns io::ErrorKind::InvalidData with the Utf8Error ('invalid utf-8 sequence'). The library is string-oriented and refuses to emit non-UTF-8 data, so binary or wrong-encoding input flowing into output triggers this error.

Source

Thrown at crates/pi-builtins/src/sed.rs:5744

	/// Flush output through a completed line when writing to a non-file stdout.
	fn flush_completed_line(&mut self) -> io::Result<()> {
		if self.line_buffered {
			#[cfg(test)]
			{
				self.low_level_flushes += 1;
			}
			self.out.flush()?;
		}
		Ok(())
	}
}

/// Implementation of the std::io::Write trait
impl Write for OutputBuffer {
	fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
		let s =
			std::str::from_utf8(buf).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;
		self.write_str(s)?;
		Ok(buf.len())
	}

	fn flush(&mut self) -> io::Result<()> {
		self.flush()
	}
}

#[cfg(unix)]
#[derive(Debug, PartialEq)]
enum WriteRange {
	Complete, // Write all specified data.
	Blocks,   // Finish write on a block boundary (to help alignment).
	None,     // No writing is needed.
}

#[cfg(unix)]

View on GitHub (pinned to 9690622007)

Solutions

1. Ensure input files are valid UTF-8: convert with iconv first (e.g. `iconv -f UTF-16 -t UTF-8 in.txt | sed ...`).
2. Do not run sed on binary data; use a byte-oriented tool or prefilter with grep -I / file to detect binary files.
3. If a multibyte character is being split, operate on whole lines/characters rather than raw byte ranges in the sed script.
4. Catch the io::Error, check `e.kind() == io::ErrorKind::InvalidData`, and surface a clear 'input must be valid UTF-8' message to the caller.
5. Validate input with std::str::from_utf8 (or `String::from_utf8`) before feeding it into sed-based processing.

Example fix

// before: feeding raw bytes from a file straight into sed output processing
let bytes = fs::read(path)?;
out.write_all(&bytes)?; // panics into InvalidData if not UTF-8

// after: validate/convert first
let text = String::from_utf8(bytes)
    .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "input is not valid UTF-8"))?;
out.write_all(text.as_bytes())?;

fn ensure_utf8(bytes: &[u8]) -> io::Result<&str> {
    std::str::from_utf8(bytes)
        .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, format!("input is not valid UTF-8: {e}")))
}

fn is_invalid_data(err: &io::Error) -> bool {
    err.kind() == io::ErrorKind::InvalidData
}

match process_result {
    Err(e) if e.kind() == io::ErrorKind::InvalidData => {
        eprintln!("sed input/output must be valid UTF-8; convert the file first (iconv) or use a byte-oriented tool");
    }
    Err(e) => return Err(e),
    Ok(v) => Ok(v),
}

When it happens

Trigger: Piping binary content (images, compressed data, UTF-16 files) through the sed builtin so that captured/processed bytes reach `OutputBuffer::write`; sed commands that concatenate or emit raw input chunks containing a truncated multibyte sequence at a chunk boundary; any `write_all`/`io::copy` path feeding non-UTF-8 bytes into the output buffer.

Common situations: Running sed on files produced on Windows in UTF-16 or Latin-1 encoding; accidentally sed-ing a binary file (e.g. `sed ... file.png`); byte-level sed operations (`y///`, character classes) splitting a multibyte UTF-8 character; concatenating script output with raw bytes from another tool.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Too many levels of symbolic links
• {}: {error}
• failed to create a unique fc temporary file
• dates before 1970 are unsupported
• missing {name}

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/5f550d7293a2daf6. Report an issue: GitHub.