{"id":"333e653fc14a53d8","repo":"pypa/pip","slug":"uncompilable-regex-tdef-0-r-in-state-state-r","errorCode":null,"errorMessage":"uncompilable regex {tdef[0]!r} in state {state!r} of {cls!r}: {err}","messagePattern":"uncompilable regex (.+?) in state (.+?) of (.+?): (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/pip/_vendor/pygments/lexer.py","lineNumber":585,"sourceCode":"                tokens.extend(cls._process_state(unprocessed, processed,\n                                                 str(tdef)))\n                continue\n            if isinstance(tdef, _inherit):\n                # should be processed already, but may not in the case of:\n                # 1. the state has no counterpart in any parent\n                # 2. the state includes more than one 'inherit'\n                continue\n            if isinstance(tdef, default):\n                new_state = cls._process_new_state(tdef.state, unprocessed, processed)\n                tokens.append((re.compile('').match, None, new_state))\n                continue\n\n            assert type(tdef) is tuple, f\"wrong rule def {tdef!r}\"\n\n            try:\n                rex = cls._process_regex(tdef[0], rflags, state)\n            except Exception as err:\n                raise ValueError(f\"uncompilable regex {tdef[0]!r} in state {state!r} of {cls!r}: {err}\") from err\n\n            token = cls._process_token(tdef[1])\n\n            if len(tdef) == 2:\n                new_state = None\n            else:\n                new_state = cls._process_new_state(tdef[2],\n                                                   unprocessed, processed)\n\n            tokens.append((rex, token, new_state))\n        return tokens\n\n    def process_tokendef(cls, name, tokendefs=None):\n        \"\"\"Preprocess a dictionary of token definitions.\"\"\"\n        processed = cls._all_tokens[name] = {}\n        tokendefs = tokendefs or cls.tokens[name]\n        for state in list(tokendefs):\n            cls._process_state(tokendefs, processed, state)","sourceCodeStart":567,"sourceCodeEnd":603,"githubUrl":"https://github.com/pypa/pip/blob/d7d0d0a39494e28ec1c407bd0680e4a4d1067791/src/pip/_vendor/pygments/lexer.py#L567-L603","documentation":"Raised by the RegexLexerMeta metaclass (in _process_state) at class-creation time when re.compile fails on a regex string found in a token rule of a custom RegexLexer subclass's `tokens` dict. The offending regex, state name, and the underlying re.error are all included.","triggerScenarios":"Defining a RegexLexer subclass whose tokens dict contains a malformed regex string (unbalanced paren, bad escape, invalid group syntax).","commonSituations":"Hand-writing lexer rules with regex typos; using a regex construct valid in the `regex` module but not stdlib `re` (e.g. named group syntax differences); copying a rule that breaks under a new Python's stricter re engine.","solutions":["Isolate the regex from the error message and test it with re.compile() directly.","Fix the syntax the embedded re.error describes (e.g. unbalanced parenthesis, bad escape \\P).","Confirm the regex is stdlib-re compatible, not just `regex`-module compatible."],"exampleFix":"# before\nclass BadLexer(RegexLexer):\n    tokens = {'root': [(r'(foo', Name)]}\n# after\nclass GoodLexer(RegexLexer):\n    tokens = {'root': [(r'(foo)', Name)]}","handlingStrategy":"validation","validationCode":"import re\ndef all_regexes_compile(tokens):\n    for state, rules in tokens.items():\n        for r in rules:\n            if isinstance(r, tuple):\n                try:\n                    re.compile(r[0])\n                except re.error as e:\n                    return False, (state, r[0], str(e))\n    return True, None","typeGuard":null,"tryCatchPattern":"try:\n    class MyLexer(RegexLexer):\n        tokens = {...}\nexcept ValueError as e:\n    if 'uncompilable regex' in str(e):\n        # fix the offending regex from the message\n        pass","preventionTips":["Compile every rule regex in isolation before assembling the lexer.","Use raw strings for rule regexes."],"tags":["pygments","lexer","regex","metaclass"],"analyzedSha":"d7d0d0a39494e28ec1c407bd0680e4a4d1067791","analyzedAt":"2026-08-04T20:55:04.259Z","schemaVersion":2}