{"record":{"id":"5f550d7293a2daf6","repo":"can1357/oh-my-pi","slug":"invalid-utf-8-sequence","errorCode":null,"errorMessage":"invalid utf-8 sequence","messagePattern":"invalid utf-8 sequence","errorType":"exception","errorClass":"io::Error","httpStatus":null,"severity":"error","filePath":"crates/pi-builtins/src/sed.rs","lineNumber":5744,"sourceCode":"\n\t/// Flush output through a completed line when writing to a non-file stdout.\n\tfn flush_completed_line(&mut self) -> io::Result<()> {\n\t\tif self.line_buffered {\n\t\t\t#[cfg(test)]\n\t\t\t{\n\t\t\t\tself.low_level_flushes += 1;\n\t\t\t}\n\t\t\tself.out.flush()?;\n\t\t}\n\t\tOk(())\n\t}\n}\n\n/// Implementation of the std::io::Write trait\nimpl Write for OutputBuffer {\n\tfn write(&mut self, buf: &[u8]) -> io::Result<usize> {\n\t\tlet s =\n\t\t\tstd::str::from_utf8(buf).map_err(|e| io::Error::new(io::ErrorKind::InvalidData, e))?;\n\t\tself.write_str(s)?;\n\t\tOk(buf.len())\n\t}\n\n\tfn flush(&mut self) -> io::Result<()> {\n\t\tself.flush()\n\t}\n}\n\n#[cfg(unix)]\n#[derive(Debug, PartialEq)]\nenum WriteRange {\n\tComplete, // Write all specified data.\n\tBlocks,   // Finish write on a block boundary (to help alignment).\n\tNone,     // No writing is needed.\n}\n\n#[cfg(unix)]","sourceCodeStart":5726,"sourceCodeEnd":5762,"githubUrl":"https://github.com/can1357/oh-my-pi/blob/969062200754ea02cfac922e5ebb8c608c079e15/crates/pi-builtins/src/sed.rs#L5726-L5762","documentation":"sed's `OutputBuffer` implements std::io::Write by converting every written byte slice with `std::str::from_utf8`; if the bytes are not valid UTF-8 it returns io::ErrorKind::InvalidData with the Utf8Error ('invalid utf-8 sequence'). The library is string-oriented and refuses to emit non-UTF-8 data, so binary or wrong-encoding input flowing into output triggers this error.","triggerScenarios":"Piping binary content (images, compressed data, UTF-16 files) through the sed builtin so that captured/processed bytes reach `OutputBuffer::write`; sed commands that concatenate or emit raw input chunks containing a truncated multibyte sequence at a chunk boundary; any `write_all`/`io::copy` path feeding non-UTF-8 bytes into the output buffer.","commonSituations":"Running sed on files produced on Windows in UTF-16 or Latin-1 encoding; accidentally sed-ing a binary file (e.g. `sed ... file.png`); byte-level sed operations (`y///`, character classes) splitting a multibyte UTF-8 character; concatenating script output with raw bytes from another tool.","solutions":["Ensure input files are valid UTF-8: convert with iconv first (e.g. `iconv -f UTF-16 -t UTF-8 in.txt | sed ...`).","Do not run sed on binary data; use a byte-oriented tool or prefilter with grep -I / file to detect binary files.","If a multibyte character is being split, operate on whole lines/characters rather than raw byte ranges in the sed script.","Catch the io::Error, check `e.kind() == io::ErrorKind::InvalidData`, and surface a clear 'input must be valid UTF-8' message to the caller.","Validate input with std::str::from_utf8 (or `String::from_utf8`) before feeding it into sed-based processing."],"exampleFix":"// before: feeding raw bytes from a file straight into sed output processing\nlet bytes = fs::read(path)?;\nout.write_all(&bytes)?; // panics into InvalidData if not UTF-8\n\n// after: validate/convert first\nlet text = String::from_utf8(bytes)\n    .map_err(|_| io::Error::new(io::ErrorKind::InvalidData, \"input is not valid UTF-8\"))?;\nout.write_all(text.as_bytes())?;","handlingStrategy":"validation","validationCode":"fn ensure_utf8(bytes: &[u8]) -> io::Result<&str> {\n    std::str::from_utf8(bytes)\n        .map_err(|e| io::Error::new(io::ErrorKind::InvalidData, format!(\"input is not valid UTF-8: {e}\")))\n}","typeGuard":"fn is_invalid_data(err: &io::Error) -> bool {\n    err.kind() == io::ErrorKind::InvalidData\n}","tryCatchPattern":"match process_result {\n    Err(e) if e.kind() == io::ErrorKind::InvalidData => {\n        eprintln!(\"sed input/output must be valid UTF-8; convert the file first (iconv) or use a byte-oriented tool\");\n    }\n    Err(e) => return Err(e),\n    Ok(v) => Ok(v),\n}","preventionTips":["Detect binary/non-UTF-8 input before processing (grep -I, file, or try from_utf8 on a sample).","Convert non-UTF-8 encodings (UTF-16, Latin-1) to UTF-8 with iconv before running sed.","Never feed binary files (images, archives) through sed-style string pipelines.","Avoid byte-level operations that can split multibyte UTF-8 characters at chunk boundaries."],"tags":["encoding","utf-8","io","sed","rust"],"backgroundTag":"invalid-utf8-sequence","analyzedSha":"969062200754ea02cfac922e5ebb8c608c079e15","analyzedAt":"2026-08-31T10:29:35.737Z","schemaVersion":2},"datasetVersion":"2026-08-31T14:17:45.589Z"}