{"record":{"id":"26f17671bf3f9c62","repo":"can1357/oh-my-pi","slug":"invalid-byte-sequence-02x","errorCode":null,"errorMessage":"invalid byte sequence: {:02x?}","messagePattern":"invalid byte sequence: (.+?)","errorType":"error_code","errorClass":"BufReadDecoderError","httpStatus":null,"severity":"error","filePath":"crates/pi-builtins/src/wc.rs","lineNumber":392,"sourceCode":"\t\t\n\t\tuse thiserror::Error;\n\t\t\n\t\tuse super::{Incomplete, str};\n\t\t\n\t\t/// Wraps a `std::io::BufRead` buffered byte stream and decode it as UTF-8.\n\t\tpub struct BufReadDecoder<B: BufRead> {\n\t\t\tbuf_read:       B,\n\t\t\tbytes_consumed: usize,\n\t\t\tincomplete:     Incomplete,\n\t\t}\n\t\t\n\t\t#[derive(Debug, Error)]\n\t\tpub enum BufReadDecoderError<'a> {\n\t\t\t/// Represents one UTF-8 error in the byte stream.\n\t\t\t///\n\t\t\t/// In lossy decoding, each such error should be replaced with U+FFFD.\n\t\t\t/// (See `BufReadDecoder::next_lossy` and `BufReadDecoderError::lossy`.)\n\t\t\t#[error(\"invalid byte sequence: {:02x?}\", .0)]\n\t\t\tInvalidByteSequence(&'a [u8]),\n\t\t\n\t\t\t/// An I/O error from the underlying byte stream\n\t\t\t#[error(\"underlying bytestream error: {}\", .0)]\n\t\t\tIo(#[source] io::Error),\n\t\t}\n\t\t\n\t\timpl<B: BufRead> BufReadDecoder<B> {\n\t\t\tpub fn new(buf_read: B) -> Self {\n\t\t\t\tSelf { buf_read, bytes_consumed: 0, incomplete: Incomplete::empty() }\n\t\t\t}\n\t\t\n\t\t\t/// Decode and consume the next chunk of UTF-8 input.\n\t\t\t///\n\t\t\t/// This method is intended to be called repeatedly until it returns `None`,\n\t\t\t/// which represents EOF from the underlying byte stream.\n\t\t\t/// This is similar to `Iterator::next`,\n\t\t\t/// except that decoded chunks borrow the decoder (~iterator)","sourceCodeStart":374,"sourceCodeEnd":410,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/crates/pi-builtins/src/wc.rs#L374-L410","documentation":"BufReadDecoderError::InvalidByteSequence reports a UTF-8 decoding error: the byte stream contained bytes that do not form valid UTF-8, and the raw offending bytes are hex-formatted via `{:02x?}`. In lossy decoding each occurrence is normally replaced with U+FFFD, so surfacing it means strict (non-lossy) decoding is in effect.","triggerScenarios":"Reading a file or stream through BufReadDecoder (used by wc) whose content is not valid UTF-8 — e.g. Latin-1/GBK text, binary data, or a truncated multi-byte sequence at a chunk boundary handled as invalid.","commonSituations":"Counting words/lines in log files written in a legacy encoding; piping binary blobs into wc; files created on Windows with CP1252 encoding; corrupted downloads.","solutions":["Re-encode the input to UTF-8 first (e.g. `iconv -f latin1 -t utf-8` into a temp file)","If byte counts suffice, use `wc -c`/byte-oriented counting that avoids UTF-8 decoding","If lossy handling is acceptable, decode with lossy mode so invalid bytes become U+FFFD instead of errors","Inspect the hex bytes in the message to identify the actual encoding"],"exampleFix":"// before: feeding Latin-1 bytes into the UTF-8 decoder -> error\n// after\nlet bytes = std::fs::read(\"input.txt\")?;\nlet (decoded, _, had_errors) = encoding_rs::LATIN_1.decode(&bytes);\n// now feed decoded (valid UTF-8) to the wc logic","handlingStrategy":"validation","validationCode":"fn is_probably_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}\n// call before counting: if !is_probably_utf8(&data) { convert or use byte mode }","typeGuard":null,"tryCatchPattern":"match result {\n    Err(BufReadDecoderError::InvalidByteSequence(bytes)) => {\n        eprintln!(\"non-UTF-8 bytes {:02x?}; re-encoding input\", bytes);\n        // re-run with iconv-converted input or lossy decoding\n    }\n    other => other.map(|_| ()),\n}","preventionTips":["Normalize all inputs to UTF-8 at ingestion (iconv/encoding_rs)","Use byte-oriented counting (-c) for binary or legacy-encoded data","Detect encoding before processing files from unknown sources","Guard pipelines so binary output never reaches text-mode wc"],"tags":["encoding","utf-8","wc","decoding"],"backgroundTag":"invalid-utf8-byte-sequence","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}