{"record":{"id":"29da669f8cf94393","repo":"cocoindex-io/cocoindex","slug":"valid-tokenizer-regex","errorCode":null,"errorMessage":"valid tokenizer regex","messagePattern":"valid tokenizer regex","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"rust/code_match/src/config.rs","lineNumber":80,"sourceCode":"            modes: ALL_MODES,\n        }\n    }\n    /// Restrict this rule to a set of lexer modes (a bitmask).\n    pub fn in_modes(mut self, modes: u8) -> Self {\n        self.modes = modes;\n        self\n    }\n}\n\n/// A position-anchored regex tokenizer (the pattern is compiled with a leading `^`).\npub struct RegexTokenizer {\n    re: Regex,\n}\n\nimpl RegexTokenizer {\n    pub fn new(pat: &str) -> Self {\n        RegexTokenizer {\n            re: Regex::new(pat).expect(\"valid tokenizer regex\"),\n        }\n    }\n}\n\nimpl Tokenizer for RegexTokenizer {\n    fn match_len(&self, input: &str) -> Option<usize> {\n        // `^` anchors at the start, so a match (if any) starts at 0.\n        self.re.find(input).map(|m| m.end()).filter(|&l| l > 0)\n    }\n}\n\n/// Convenience: a regex-based rule.\npub fn regex_rule(pat: &str, kind: TokKind) -> TokenRule {\n    TokenRule::new(RegexTokenizer::new(pat), kind)\n}\n\n// --- shared (generic) token-class builders, composed by the language modules ---\n","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/cocoindex-io/cocoindex/blob/e84aa99b3292c5270a4b313b2a7137ad9ce8ab3b/rust/code_match/src/config.rs#L62-L98","documentation":"RegexTokenizer::new panics immediately if the tokenizer pattern string is not a valid regex for the Rust regex crate. The panic message 'valid tokenizer regex' signals that a developer-supplied tokenizer pattern failed to compile. This fails fast at construction rather than during matching.","triggerScenarios":"Constructing RegexTokenizer::new(\"...\") with an invalid pattern: unbalanced groups/brackets, invalid escape sequences (e.g. \\y), unsupported backreferences/lookaround (not supported by the regex crate), or empty pattern semantics the crate rejects.","commonSituations":"Porting a JavaScript/PCRE tokenizer pattern into Rust config (lookaheads `(?=...)` and backrefs `\\1` are unsupported); typos like `[a-z` ; users pasting glob patterns instead of regex.","solutions":["Test the pattern with regex crate syntax first (e.g. regex-cli or a unit test calling Regex::new).","Remove PCRE-only features (lookahead/lookbehind, backreferences) — rewrite the tokenizer logic instead.","Fix bracket/group balance and escape metacharacters for literal tokens.","Prefer a simpler pattern split into multiple simpler tokenizers if the combined one is too complex."],"exampleFix":"// before\nRegexTokenizer::new(\"\\\\w+|(?=\\\\s)\") // lookahead unsupported\n\n// after\nRegexTokenizer::new(\"\\\\w+|\\\\s+\")","handlingStrategy":"validation","validationCode":"// fail early with a real error instead of panicking\nfn try_tokenizer(pat: &str) -> Result<RegexTokenizer, regex::Error> {\n    regex::Regex::new(pat)?;\n    Ok(RegexTokenizer::new(pat))\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Verify patterns against Rust `regex` crate syntax (no lookaround/backreferences).","Add a constructor test for every custom tokenizer pattern.","Validate user-supplied patterns with Regex::new before passing to RegexTokenizer::new."],"tags":["rust","regex","panic","tokenizer"],"backgroundTag":"invalid-regex-pattern","analyzedSha":"e84aa99b3292c5270a4b313b2a7137ad9ce8ab3b","analyzedAt":"2026-09-08T15:59:19.997Z","contentChangedAt":"2026-09-08T15:59:19.997Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}