pypa/pip · error · ValueError

uncompilable regex {tdef[0]!r} in state {state!r} of {cls!r}

Error message

uncompilable regex {tdef[0]!r} in state {state!r} of {cls!r}: {err}

What it means

Raised by the RegexLexerMeta metaclass (in _process_state) at class-creation time when re.compile fails on a regex string found in a token rule of a custom RegexLexer subclass's `tokens` dict. The offending regex, state name, and the underlying re.error are all included.

Source

Thrown at src/pip/_vendor/pygments/lexer.py:585

                tokens.extend(cls._process_state(unprocessed, processed,
                                                 str(tdef)))
                continue
            if isinstance(tdef, _inherit):
                # should be processed already, but may not in the case of:
                # 1. the state has no counterpart in any parent
                # 2. the state includes more than one 'inherit'
                continue
            if isinstance(tdef, default):
                new_state = cls._process_new_state(tdef.state, unprocessed, processed)
                tokens.append((re.compile('').match, None, new_state))
                continue

            assert type(tdef) is tuple, f"wrong rule def {tdef!r}"

            try:
                rex = cls._process_regex(tdef[0], rflags, state)
            except Exception as err:
                raise ValueError(f"uncompilable regex {tdef[0]!r} in state {state!r} of {cls!r}: {err}") from err

            token = cls._process_token(tdef[1])

            if len(tdef) == 2:
                new_state = None
            else:
                new_state = cls._process_new_state(tdef[2],
                                                   unprocessed, processed)

            tokens.append((rex, token, new_state))
        return tokens

    def process_tokendef(cls, name, tokendefs=None):
        """Preprocess a dictionary of token definitions."""
        processed = cls._all_tokens[name] = {}
        tokendefs = tokendefs or cls.tokens[name]
        for state in list(tokendefs):
            cls._process_state(tokendefs, processed, state)

View on GitHub (pinned to d7d0d0a394)

Solutions

1. Isolate the regex from the error message and test it with re.compile() directly.
2. Fix the syntax the embedded re.error describes (e.g. unbalanced parenthesis, bad escape \P).
3. Confirm the regex is stdlib-re compatible, not just `regex`-module compatible.

Example fix

# before
class BadLexer(RegexLexer):
    tokens = {'root': [(r'(foo', Name)]}
# after
class GoodLexer(RegexLexer):
    tokens = {'root': [(r'(foo)', Name)]}

import re
def all_regexes_compile(tokens):
    for state, rules in tokens.items():
        for r in rules:
            if isinstance(r, tuple):
                try:
                    re.compile(r[0])
                except re.error as e:
                    return False, (state, r[0], str(e))
    return True, None

try:
    class MyLexer(RegexLexer):
        tokens = {...}
except ValueError as e:
    if 'uncompilable regex' in str(e):
        # fix the offending regex from the message
        pass

When it happens

Trigger: Defining a RegexLexer subclass whose tokens dict contains a malformed regex string (unbalanced paren, bad escape, invalid group syntax).

Common situations: Hand-writing lexer rules with regex typos; using a regex construct valid in the `regex` module but not stdlib `re` (e.g. named group syntax differences); copying a rule that breaks under a new Python's stricter re engine.

Related errors

• No such group
• lex() argument must be a lexer instance, not a class
• no lexer for alias {_alias!r} found
• no valid {lexername} class found in {filename}
• cannot read {filename}: {err}

AI-assisted analysis of pypa/pip@d7d0d0a394 (2026-08-04). Data as JSON: /data/errors/333e653fc14a53d8.json. Report an issue: GitHub.