What causes "invalid byte sequence: {:02x?}" in can1357/oh-my-pi?

Re-encode the input to UTF-8 first (e.g. `iconv -f latin1 -t utf-8` into a temp file)

can1357/oh-my-pi · error · BufReadDecoderError

invalid byte sequence: {:02x?}

Error message

invalid byte sequence: {:02x?}

What it means

BufReadDecoderError::InvalidByteSequence reports a UTF-8 decoding error: the byte stream contained bytes that do not form valid UTF-8, and the raw offending bytes are hex-formatted via `{:02x?}`. In lossy decoding each occurrence is normally replaced with U+FFFD, so surfacing it means strict (non-lossy) decoding is in effect.

Source

Thrown at crates/pi-builtins/src/wc.rs:392

		
		use thiserror::Error;
		
		use super::{Incomplete, str};
		
		/// Wraps a `std::io::BufRead` buffered byte stream and decode it as UTF-8.
		pub struct BufReadDecoder<B: BufRead> {
			buf_read:       B,
			bytes_consumed: usize,
			incomplete:     Incomplete,
		}
		
		#[derive(Debug, Error)]
		pub enum BufReadDecoderError<'a> {
			/// Represents one UTF-8 error in the byte stream.
			///
			/// In lossy decoding, each such error should be replaced with U+FFFD.
			/// (See `BufReadDecoder::next_lossy` and `BufReadDecoderError::lossy`.)
			#[error("invalid byte sequence: {:02x?}", .0)]
			InvalidByteSequence(&'a [u8]),
		
			/// An I/O error from the underlying byte stream
			#[error("underlying bytestream error: {}", .0)]
			Io(#[source] io::Error),
		}
		
		impl<B: BufRead> BufReadDecoder<B> {
			pub fn new(buf_read: B) -> Self {
				Self { buf_read, bytes_consumed: 0, incomplete: Incomplete::empty() }
			}
		
			/// Decode and consume the next chunk of UTF-8 input.
			///
			/// This method is intended to be called repeatedly until it returns `None`,
			/// which represents EOF from the underlying byte stream.
			/// This is similar to `Iterator::next`,
			/// except that decoded chunks borrow the decoder (~iterator)

View on GitHub (pinned to 9690622007)

Solutions

Re-encode the input to UTF-8 first (e.g. `iconv -f latin1 -t utf-8` into a temp file)
If byte counts suffice, use `wc -c`/byte-oriented counting that avoids UTF-8 decoding
If lossy handling is acceptable, decode with lossy mode so invalid bytes become U+FFFD instead of errors
Inspect the hex bytes in the message to identify the actual encoding

Example fix

// before: feeding Latin-1 bytes into the UTF-8 decoder -> error
// after
let bytes = std::fs::read("input.txt")?;
let (decoded, _, had_errors) = encoding_rs::LATIN_1.decode(&bytes);
// now feed decoded (valid UTF-8) to the wc logic

Defensive patterns

Strategy: validation

Validate before calling

fn is_probably_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}
// call before counting: if !is_probably_utf8(&data) { convert or use byte mode }

Try / catch

match result {
    Err(BufReadDecoderError::InvalidByteSequence(bytes)) => {
        eprintln!("non-UTF-8 bytes {:02x?}; re-encoding input", bytes);
        // re-run with iconv-converted input or lossy decoding
    }
    other => other.map(|_| ()),
}

Prevention

Normalize all inputs to UTF-8 at ingestion (iconv/encoding_rs)
Use byte-oriented counting (-c) for binary or legacy-encoded data
Detect encoding before processing files from unknown sources
Guard pipelines so binary output never reaches text-mode wc

When it happens

Trigger: Reading a file or stream through BufReadDecoder (used by wc) whose content is not valid UTF-8 — e.g. Latin-1/GBK text, binary data, or a truncated multi-byte sequence at a chunk boundary handled as invalid.

Common situations: Counting words/lines in log files written in a legacy encoding; piping binary blobs into wc; files created on Windows with CP1252 encoding; corrupted downloads.

Related errors

AI-assisted analysis of can1357/oh-my-pi@9690622007 (2026-08-31). Data as JSON: /api/errors/26f17671bf3f9c62. Report an issue: GitHub.