{"record":{"id":"c73743d15d2625fc","repo":"antlr/antlr4","slug":"unknown-token-s-in-pattern-s","errorCode":null,"errorMessage":"Unknown token %s in pattern: %s","messagePattern":"Unknown token (.+?) in pattern: (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"runtime/Python3/src/antlr4/tree/ParseTreePatternMatcher.py","lineNumber":287,"sourceCode":"            if tree.getChildCount()==1 and isinstance(tree.getChild(0), TerminalNode ):\n                c = tree.getChild(0)\n                if isinstance( c.symbol, RuleTagToken ):\n                    return c.symbol\n        return None\n\n    def tokenize(self, pattern:str):\n        # split pattern into chunks: sea (raw input) and islands (<ID>, <expr>)\n        chunks = self.split(pattern)\n\n        # create token stream from text and tags\n        tokens = list()\n        for chunk in chunks:\n            if isinstance( chunk, TagChunk ):\n                # add special rule token or conjure up new token from name\n                if chunk.tag[0].isupper():\n                    ttype = self.parser.getTokenType(chunk.tag)\n                    if ttype==Token.INVALID_TYPE:\n                        raise Exception(\"Unknown token \" + str(chunk.tag) + \" in pattern: \" + pattern)\n                    tokens.append(TokenTagToken(chunk.tag, ttype, chunk.label))\n                elif chunk.tag[0].islower():\n                    ruleIndex = self.parser.getRuleIndex(chunk.tag)\n                    if ruleIndex==-1:\n                        raise Exception(\"Unknown rule \" + str(chunk.tag) + \" in pattern: \" + pattern)\n                    ruleImaginaryTokenType = self.parser.getATNWithBypassAlts().ruleToTokenType[ruleIndex]\n                    tokens.append(RuleTagToken(chunk.tag, ruleImaginaryTokenType, chunk.label))\n                else:\n                    raise Exception(\"invalid tag: \" + str(chunk.tag) + \" in pattern: \" + pattern)\n            else:\n                self.lexer.setInputStream(InputStream(chunk.text))\n                t = self.lexer.nextToken()\n                while t.type!=Token.EOF:\n                    tokens.append(t)\n                    t = self.lexer.nextToken()\n        return tokens\n\n    # Split {@code <ID> = <e:expr> ;} into 4 chunks for tokenizing by {@link #tokenize}.#","sourceCodeStart":269,"sourceCodeEnd":305,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Python3/src/antlr4/tree/ParseTreePatternMatcher.py#L269-L305","documentation":"Raised by ParseTreePatternMatcher.tokenize() when a tag starting with an uppercase letter is not a known token name in the parser. Uppercase-first tags are token tags (e.g. <ID>); the matcher resolves them with parser.getTokenType() and InvalidType means the grammar has no such token.","triggerScenarios":"Compiling a pattern containing a tag like <ID>, <STRING>, or <myLabel:TOKEN> where that token name does not exist in the parser's vocabulary. Also triggered when a rule tag is misspelled with a capital first letter (e.g. <Expr> instead of <expr>).","commonSituations":"Pattern written for a different grammar, renamed tokens after a grammar change, or case confusion between token names (uppercase) and rule names (lowercase).","solutions":["Check the tag spelling against the parser's token names (parser.symbolicNames)","If the tag refers to a parser rule, lowercase the first letter (<expr>)","If the token genuinely exists, verify you compiled the pattern with the matching generated parser instance"],"exampleFix":"# before: token is ID in the grammar\npattern = parser.compilePattern('<IDENT> + <IDENT>', 0)\n\n# after\npattern = parser.compilePattern('<ID> + <ID>', 0)","handlingStrategy":"validation","validationCode":"tag_names = {t for t in parser.symbolicNames if t}\nassert 'ID' in tag_names, 'unknown token tag'\npattern = parser.compilePattern('<ID>', 0)","typeGuard":"def is_known_token(tag: str, parser) -> bool:\n    return parser.getTokenType(tag) != -1  # Token.INVALID_TYPE","tryCatchPattern":"try:\n    p = matcher.compileTreePattern(pat, idx)\nexcept Exception as e:\n    if str(e).startswith('Unknown token'):\n        # unknown token tag: check parser.symbolicNames\n        ...","preventionTips":["Cross-check uppercase tags against parser.symbolicNames","Lowercase-first letters for rule tags"],"tags":["antlr","python","parse-tree-matching","pattern-compile"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}