{"record":{"id":"4b66cfc16af718d2","repo":"antlr/antlr4","slug":"unknown-token-tag-in-pattern-pattern","errorCode":null,"errorMessage":"Unknown token {tag} in pattern: {pattern}","messagePattern":"Unknown token (.+?) in pattern: (.+?)","errorType":"validation","errorClass":"ArgumentException","httpStatus":null,"severity":"error","filePath":"runtime/CSharp/src/Tree/Pattern/ParseTreePatternMatcher.cs","lineNumber":540,"sourceCode":"\n        public virtual IList<IToken> Tokenize(string pattern)\n        {\n            // split pattern into chunks: sea (raw input) and islands (<ID>, <expr>)\n            IList<Chunk> chunks = Split(pattern);\n            // create token stream from text and tags\n            IList<IToken> tokens = new List<IToken>();\n            foreach (Chunk chunk in chunks)\n            {\n                if (chunk is TagChunk)\n                {\n                    TagChunk tagChunk = (TagChunk)chunk;\n                    // add special rule token or conjure up new token from name\n                    if (System.Char.IsUpper(tagChunk.Tag[0]))\n                    {\n                        int ttype = parser.GetTokenType(tagChunk.Tag);\n                        if (ttype == TokenConstants.InvalidType)\n                        {\n                            throw new ArgumentException(\"Unknown token \" + tagChunk.Tag + \" in pattern: \" + pattern);\n                        }\n                        TokenTagToken t = new TokenTagToken(tagChunk.Tag, ttype, tagChunk.Label);\n                        tokens.Add(t);\n                    }\n                    else\n                    {\n                        if (System.Char.IsLower(tagChunk.Tag[0]))\n                        {\n                            int ruleIndex = parser.GetRuleIndex(tagChunk.Tag);\n                            if (ruleIndex == -1)\n                            {\n                                throw new ArgumentException(\"Unknown rule \" + tagChunk.Tag + \" in pattern: \" + pattern);\n                            }\n                            int ruleImaginaryTokenType = parser.GetATNWithBypassAlts().ruleToTokenType[ruleIndex];\n                            tokens.Add(new RuleTagToken(tagChunk.Tag, ruleImaginaryTokenType, tagChunk.Label));\n                        }\n                        else\n                        {","sourceCodeStart":522,"sourceCodeEnd":558,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/CSharp/src/Tree/Pattern/ParseTreePatternMatcher.cs#L522-L558","documentation":"While compiling a parse-tree pattern, the matcher found a tag starting with an uppercase letter (token tag, e.g. <ID>) whose name is not a known token name in the parser's vocabulary, so GetTokenType returned TokenConstants.InvalidType. This is a pattern-authoring error: the tag must name a token declared in the grammar used to build the parser.","triggerScenarios":"Calling parser.CompileParseTreePattern(\"<IDENT> = <expr>\", ...) when the grammar's token is ID, not IDENT; using a parser built from a different/newer grammar where the token was renamed or removed; typos in the token tag.","commonSituations":"Grammar evolves between versions (token renamed in the .g4 and regenerated parser, patterns not updated); patterns loaded from external files/config authored against an older grammar; copy-pasting example patterns from another grammar.","solutions":["Fix the tag to match an actual token name from the grammar (check parser.Vocabulary or the generated *.tokens file).","If the tag refers to a parser rule, lowercase it so it is treated as a rule tag instead of a token tag.","After grammar changes, regenerate the parser and update all pattern strings in lockstep; keep patterns near the grammar or generate them from the vocabulary.","Log parser.Vocabulary.GetSymbolicNames() at startup to validate configured patterns against the live vocabulary."],"exampleFix":"// before\nvar p = parser.CompileParseTreePattern(\"<IDENT> = <expr>\", ExprParser.RULE_stmt, null);\n\n// after (token is named ID in the grammar)\nvar p = parser.CompileParseTreePattern(\"<ID> = <expr>\", ExprParser.RULE_stmt, null);","handlingStrategy":"validation","validationCode":"// Validate token tags against the live vocabulary before compiling\nforeach (var tag in ExtractTokenTags(pattern))\n    if (parser.GetTokenType(tag) == TokenConstants.InvalidType)\n        throw new ArgumentException($\"pattern references unknown token <{tag}>\");\nvar p = parser.CompileParseTreePattern(pattern, ruleIndex, null);","typeGuard":null,"tryCatchPattern":"try { var p = parser.CompileParseTreePattern(pattern, ruleIndex, null); } catch (ArgumentException ex) when (ex.Message.StartsWith(\"Unknown token\")) { /* report pattern and vocabulary to the user */ }","preventionTips":["Keep pattern strings in the same change unit as the grammar; update both on token renames.","Validate patterns against parser.Vocabulary at startup when they come from config."],"tags":["antlr","csharp","pattern","vocabulary","grammar-mismatch"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}