antlr/antlr4 · error · Exception

Unknown token in pattern

Error message

Unknown token %s in pattern: %s

What it means

Raised by ParseTreePatternMatcher.tokenize() when a tag starting with an uppercase letter is not a known token name in the parser. Uppercase-first tags are token tags (e.g. <ID>); the matcher resolves them with parser.getTokenType() and InvalidType means the grammar has no such token.

Solutions

1. Check the tag spelling against the parser's token names (parser.symbolicNames)
2. If the tag refers to a parser rule, lowercase the first letter (<expr>)
3. If the token genuinely exists, verify you compiled the pattern with the matching generated parser instance

Example fix

# before: token is ID in the grammar
pattern = parser.compilePattern('<IDENT> + <IDENT>', 0)

# after
pattern = parser.compilePattern('<ID> + <ID>', 0)

tag_names = {t for t in parser.symbolicNames if t}
assert 'ID' in tag_names, 'unknown token tag'
pattern = parser.compilePattern('<ID>', 0)

def is_known_token(tag: str, parser) -> bool:
    return parser.getTokenType(tag) != -1  # Token.INVALID_TYPE

try:
    p = matcher.compileTreePattern(pat, idx)
except Exception as e:
    if str(e).startswith('Unknown token'):
        # unknown token tag: check parser.symbolicNames
        ...

When it happens

Trigger: Compiling a pattern containing a tag like <ID>, <STRING>, or <myLabel:TOKEN> where that token name does not exist in the parser's vocabulary. Also triggered when a rule tag is misspelled with a capital first letter (e.g. <Expr> instead of <expr>).

Common situations: Pattern written for a different grammar, renamed tokens after a grammar change, or case confusion between token names (uppercase) and rule names (lowercase).

Related errors

• invalid tag: in pattern
• missing start tag in pattern
• StartRuleDoesNotConsumeFullPattern
• tag delimiters out of order in pattern
• Unknown rule in pattern

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/c73743d15d2625fc. Report an issue: GitHub.

Appendix: source

Thrown at runtime/Python3/src/antlr4/tree/ParseTreePatternMatcher.py:287

            if tree.getChildCount()==1 and isinstance(tree.getChild(0), TerminalNode ):
                c = tree.getChild(0)
                if isinstance( c.symbol, RuleTagToken ):
                    return c.symbol
        return None

    def tokenize(self, pattern:str):
        # split pattern into chunks: sea (raw input) and islands (<ID>, <expr>)
        chunks = self.split(pattern)

        # create token stream from text and tags
        tokens = list()
        for chunk in chunks:
            if isinstance( chunk, TagChunk ):
                # add special rule token or conjure up new token from name
                if chunk.tag[0].isupper():
                    ttype = self.parser.getTokenType(chunk.tag)
                    if ttype==Token.INVALID_TYPE:
                        raise Exception("Unknown token " + str(chunk.tag) + " in pattern: " + pattern)
                    tokens.append(TokenTagToken(chunk.tag, ttype, chunk.label))
                elif chunk.tag[0].islower():
                    ruleIndex = self.parser.getRuleIndex(chunk.tag)
                    if ruleIndex==-1:
                        raise Exception("Unknown rule " + str(chunk.tag) + " in pattern: " + pattern)
                    ruleImaginaryTokenType = self.parser.getATNWithBypassAlts().ruleToTokenType[ruleIndex]
                    tokens.append(RuleTagToken(chunk.tag, ruleImaginaryTokenType, chunk.label))
                else:
                    raise Exception("invalid tag: " + str(chunk.tag) + " in pattern: " + pattern)
            else:
                self.lexer.setInputStream(InputStream(chunk.text))
                t = self.lexer.nextToken()
                while t.type!=Token.EOF:
                    tokens.append(t)
                    t = self.lexer.nextToken()
        return tokens

    # Split {@code <ID> = <e:expr> ;} into 4 chunks for tokenizing by {@link #tokenize}.#

View on GitHub (pinned to 7d5770395b)