antlr/antlr4 · error · ArgumentException

Unknown token {tag} in pattern: {pattern}

Error message

Unknown token {tag} in pattern: {pattern}

What it means

While compiling a parse-tree pattern, the matcher found a tag starting with an uppercase letter (token tag, e.g. <ID>) whose name is not a known token name in the parser's vocabulary, so GetTokenType returned TokenConstants.InvalidType. This is a pattern-authoring error: the tag must name a token declared in the grammar used to build the parser.

Source

Thrown at runtime/CSharp/src/Tree/Pattern/ParseTreePatternMatcher.cs:540

        public virtual IList<IToken> Tokenize(string pattern)
        {
            // split pattern into chunks: sea (raw input) and islands (<ID>, <expr>)
            IList<Chunk> chunks = Split(pattern);
            // create token stream from text and tags
            IList<IToken> tokens = new List<IToken>();
            foreach (Chunk chunk in chunks)
            {
                if (chunk is TagChunk)
                {
                    TagChunk tagChunk = (TagChunk)chunk;
                    // add special rule token or conjure up new token from name
                    if (System.Char.IsUpper(tagChunk.Tag[0]))
                    {
                        int ttype = parser.GetTokenType(tagChunk.Tag);
                        if (ttype == TokenConstants.InvalidType)
                        {
                            throw new ArgumentException("Unknown token " + tagChunk.Tag + " in pattern: " + pattern);
                        }
                        TokenTagToken t = new TokenTagToken(tagChunk.Tag, ttype, tagChunk.Label);
                        tokens.Add(t);
                    }
                    else
                    {
                        if (System.Char.IsLower(tagChunk.Tag[0]))
                        {
                            int ruleIndex = parser.GetRuleIndex(tagChunk.Tag);
                            if (ruleIndex == -1)
                            {
                                throw new ArgumentException("Unknown rule " + tagChunk.Tag + " in pattern: " + pattern);
                            }
                            int ruleImaginaryTokenType = parser.GetATNWithBypassAlts().ruleToTokenType[ruleIndex];
                            tokens.Add(new RuleTagToken(tagChunk.Tag, ruleImaginaryTokenType, tagChunk.Label));
                        }
                        else
                        {

View on GitHub (pinned to 7d5770395b)

Solutions

1. Fix the tag to match an actual token name from the grammar (check parser.Vocabulary or the generated *.tokens file).
2. If the tag refers to a parser rule, lowercase it so it is treated as a rule tag instead of a token tag.
3. After grammar changes, regenerate the parser and update all pattern strings in lockstep; keep patterns near the grammar or generate them from the vocabulary.
4. Log parser.Vocabulary.GetSymbolicNames() at startup to validate configured patterns against the live vocabulary.

Example fix

// before
var p = parser.CompileParseTreePattern("<IDENT> = <expr>", ExprParser.RULE_stmt, null);

// after (token is named ID in the grammar)
var p = parser.CompileParseTreePattern("<ID> = <expr>", ExprParser.RULE_stmt, null);

// Validate token tags against the live vocabulary before compiling
foreach (var tag in ExtractTokenTags(pattern))
    if (parser.GetTokenType(tag) == TokenConstants.InvalidType)
        throw new ArgumentException($"pattern references unknown token <{tag}>");
var p = parser.CompileParseTreePattern(pattern, ruleIndex, null);

try { var p = parser.CompileParseTreePattern(pattern, ruleIndex, null); } catch (ArgumentException ex) when (ex.Message.StartsWith("Unknown token")) { /* report pattern and vocabulary to the user */ }

When it happens

Trigger: Calling parser.CompileParseTreePattern("<IDENT> = <expr>", ...) when the grammar's token is ID, not IDENT; using a parser built from a different/newer grammar where the token was renamed or removed; typos in the token tag.

Common situations: Grammar evolves between versions (token renamed in the .g4 and regenerated parser, patterns not updated); patterns loaded from external files/config authored against an older grammar; copy-pasting example patterns from another grammar.

Related errors

• Unknown rule {tag} in pattern: {pattern}
• pattern cannot be null
• invalid tag: {tag} in pattern: {pattern}
• unterminated tag in pattern: {pattern}
• missing start tag in pattern: {pattern}

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/4b66cfc16af718d2. Report an issue: GitHub.