What causes "valid tokenizer regex" in cocoindex-io/cocoindex?

Test the pattern with regex crate syntax first (e.g. regex-cli or a unit test calling Regex::new).

valid tokenizer regex

Error message

valid tokenizer regex

What it means

RegexTokenizer::new panics immediately if the tokenizer pattern string is not a valid regex for the Rust regex crate. The panic message 'valid tokenizer regex' signals that a developer-supplied tokenizer pattern failed to compile. This fails fast at construction rather than during matching.

Source

Thrown at rust/code_match/src/config.rs:80

            modes: ALL_MODES,
        }
    }
    /// Restrict this rule to a set of lexer modes (a bitmask).
    pub fn in_modes(mut self, modes: u8) -> Self {
        self.modes = modes;
        self
    }
}

/// A position-anchored regex tokenizer (the pattern is compiled with a leading `^`).
pub struct RegexTokenizer {
    re: Regex,
}

impl RegexTokenizer {
    pub fn new(pat: &str) -> Self {
        RegexTokenizer {
            re: Regex::new(pat).expect("valid tokenizer regex"),
        }
    }
}

impl Tokenizer for RegexTokenizer {
    fn match_len(&self, input: &str) -> Option<usize> {
        // `^` anchors at the start, so a match (if any) starts at 0.
        self.re.find(input).map(|m| m.end()).filter(|&l| l > 0)
    }
}

/// Convenience: a regex-based rule.
pub fn regex_rule(pat: &str, kind: TokKind) -> TokenRule {
    TokenRule::new(RegexTokenizer::new(pat), kind)
}

// --- shared (generic) token-class builders, composed by the language modules ---

View on GitHub (pinned to e84aa99b32)

Solutions

Test the pattern with regex crate syntax first (e.g. regex-cli or a unit test calling Regex::new).
Remove PCRE-only features (lookahead/lookbehind, backreferences) — rewrite the tokenizer logic instead.
Fix bracket/group balance and escape metacharacters for literal tokens.
Prefer a simpler pattern split into multiple simpler tokenizers if the combined one is too complex.

Example fix

// before
RegexTokenizer::new("\\w+|(?=\\s)") // lookahead unsupported

// after
RegexTokenizer::new("\\w+|\\s+")

Defensive patterns

Strategy: validation

Validate before calling

// fail early with a real error instead of panicking
fn try_tokenizer(pat: &str) -> Result<RegexTokenizer, regex::Error> {
    regex::Regex::new(pat)?;
    Ok(RegexTokenizer::new(pat))
}

Prevention

Verify patterns against Rust `regex` crate syntax (no lookaround/backreferences).
Add a constructor test for every custom tokenizer pattern.
Validate user-supplied patterns with Regex::new before passing to RegexTokenizer::new.

When it happens

Trigger: Constructing RegexTokenizer::new("...") with an invalid pattern: unbalanced groups/brackets, invalid escape sequences (e.g. \y), unsupported backreferences/lookaround (not supported by the regex crate), or empty pattern semantics the crate rejects.

Common situations: Porting a JavaScript/PCRE tokenizer pattern into Rust config (lookaheads `(?=...)` and backrefs `\1` are unsupported); typos like `[a-z` ; users pasting glob patterns instead of regex.

Related errors

AI-assisted analysis of cocoindex-io/cocoindex@e84aa99b32 (2026-09-08). Data as JSON: /api/errors/29da669f8cf94393. Report an issue: GitHub.