{"id":"8ffbcb24ce27cbc4","repo":"BurntSushi/ripgrep","slug":"found-invalid-utf-8-in-pattern-at-byte-offset","errorCode":null,"errorMessage":"found invalid UTF-8 in pattern at byte offset {}: {} (disable Unicode mode and use hex escape sequences to match arbitrary bytes in a pattern, e.g., '(?-u)\\xFF')","messagePattern":"found invalid UTF-8 in pattern at byte offset (.+?): (.+?) \\(disable Unicode mode and use hex escape sequences to match arbitrary bytes in a pattern, e\\.g\\., '\\(\\?-u\\)\\\\xFF'\\)","errorType":"validation","errorClass":"InvalidPatternError","httpStatus":null,"severity":"error","filePath":"crates/cli/src/pattern.rs","lineNumber":29,"sourceCode":"#[derive(Clone, Debug, Eq, PartialEq)]\npub struct InvalidPatternError {\n    original: String,\n    valid_up_to: usize,\n}\n\nimpl InvalidPatternError {\n    /// Returns the index in the given string up to which valid UTF-8 was\n    /// verified.\n    pub fn valid_up_to(&self) -> usize {\n        self.valid_up_to\n    }\n}\n\nimpl std::error::Error for InvalidPatternError {}\n\nimpl std::fmt::Display for InvalidPatternError {\n    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n        write!(\n            f,\n            \"found invalid UTF-8 in pattern at byte offset {}: {} \\\n             (disable Unicode mode and use hex escape sequences to match \\\n             arbitrary bytes in a pattern, e.g., '(?-u)\\\\xFF')\",\n            self.valid_up_to, self.original,\n        )\n    }\n}\n\nimpl From<InvalidPatternError> for io::Error {\n    fn from(paterr: InvalidPatternError) -> io::Error {\n        io::Error::new(io::ErrorKind::Other, paterr)\n    }\n}\n\n/// Convert an OS string into a regular expression pattern.\n///\n/// This conversion fails if the given pattern is not valid UTF-8, in which","sourceCodeStart":11,"sourceCodeEnd":47,"githubUrl":"https://github.com/BurntSushi/ripgrep/blob/3fce3b5bb0236da2df6d99672afb8a719642eca7/crates/cli/src/pattern.rs#L11-L47","documentation":"InvalidPatternError is returned by pattern_from_os / pattern_from_bytes when a regex pattern supplied by the user is not valid UTF-8. The Display message reports the byte offset where decoding failed and suggests disabling Unicode mode and using hex escapes like (?-u)\\xFF so the regex engine can match the raw bytes instead. This gives end users an actionable, targeted error rather than a generic 'invalid utf8'.","triggerScenarios":"A command-line argument pattern (OsStr) or a bytes pattern read from a file/stdin contains non-UTF-8 bytes; pattern_from_os/to_str() or pattern_from_bytes/std::str::from_utf8 returns Err.","commonSituations":"Searching binary logs with a pattern containing high bytes; patterns read from a file with CRLF or legacy encodings; shell that passes through raw bytes; patterns with a stray 0xFF from a copy-paste error.","solutions":["Rewrite the pattern to be valid UTF-8.","If you genuinely need to match arbitrary bytes, disable Unicode mode and use hex escapes: (?-u)\\xFF in place of the offending byte.","Sanitize/normalize the pattern source (strip invalid bytes or transcode to UTF-8) before passing it to the parser."],"exampleFix":"// before\nlet pat = pattern_from_bytes(b\"abc\\xFFxyz\")?; // InvalidPatternError\n\n// after\nlet pat = r\"(?-u)abc\\xFFxyz\"; // valid UTF-8 regex matching the same bytes","handlingStrategy":"validation","validationCode":"use std::str;\nfn is_valid_utf8_pattern(b: &[u8]) -> bool {\n    str::from_utf8(b).is_ok()\n}","typeGuard":"fn valid_utf8_pattern(b: &[u8]) -> Option<&str> {\n    str::from_utf8(b).ok()\n}","tryCatchPattern":"let pat = match pattern_from_bytes(raw) {\n    Ok(p) => p,\n    Err(e) => { eprintln!(\"{e}\"); return; }\n};","preventionTips":["Validate patterns as UTF-8 before passing to the regex builder.","For binary patterns, build the regex with Unicode disabled and hex escapes from the start.","Normalize pattern sources (files/stdin) to UTF-8 on ingest."],"tags":["pattern","utf-8","regex","cli"],"analyzedSha":"3fce3b5bb0236da2df6d99672afb8a719642eca7","analyzedAt":"2026-08-06T01:39:05.073Z","schemaVersion":2}