antlr/antlr4 · error · Exception

Unknown token %s in pattern: %s

Error message

Unknown token %s in pattern: %s

What it means

Raised by ParseTreePatternMatcher.tokenize() when a tag starting with an uppercase letter is not a known token name in the parser. Uppercase-first tags are token tags (e.g. <ID>); the matcher resolves them with parser.getTokenType() and InvalidType means the grammar has no such token.

Source

Thrown at runtime/Python3/src/antlr4/tree/ParseTreePatternMatcher.py:287

            if tree.getChildCount()==1 and isinstance(tree.getChild(0), TerminalNode ):
                c = tree.getChild(0)
                if isinstance( c.symbol, RuleTagToken ):
                    return c.symbol
        return None

    def tokenize(self, pattern:str):
        # split pattern into chunks: sea (raw input) and islands (<ID>, <expr>)
        chunks = self.split(pattern)

        # create token stream from text and tags
        tokens = list()
        for chunk in chunks:
            if isinstance( chunk, TagChunk ):
                # add special rule token or conjure up new token from name
                if chunk.tag[0].isupper():
                    ttype = self.parser.getTokenType(chunk.tag)
                    if ttype==Token.INVALID_TYPE:
                        raise Exception("Unknown token " + str(chunk.tag) + " in pattern: " + pattern)
                    tokens.append(TokenTagToken(chunk.tag, ttype, chunk.label))
                elif chunk.tag[0].islower():
                    ruleIndex = self.parser.getRuleIndex(chunk.tag)
                    if ruleIndex==-1:
                        raise Exception("Unknown rule " + str(chunk.tag) + " in pattern: " + pattern)
                    ruleImaginaryTokenType = self.parser.getATNWithBypassAlts().ruleToTokenType[ruleIndex]
                    tokens.append(RuleTagToken(chunk.tag, ruleImaginaryTokenType, chunk.label))
                else:
                    raise Exception("invalid tag: " + str(chunk.tag) + " in pattern: " + pattern)
            else:
                self.lexer.setInputStream(InputStream(chunk.text))
                t = self.lexer.nextToken()
                while t.type!=Token.EOF:
                    tokens.append(t)
                    t = self.lexer.nextToken()
        return tokens

    # Split {@code <ID> = <e:expr> ;} into 4 chunks for tokenizing by {@link #tokenize}.#

View on GitHub (pinned to 7d5770395b)

Solutions

  1. Check the tag spelling against the parser's token names (parser.symbolicNames)
  2. If the tag refers to a parser rule, lowercase the first letter (<expr>)
  3. If the token genuinely exists, verify you compiled the pattern with the matching generated parser instance

Example fix

# before: token is ID in the grammar
pattern = parser.compilePattern('<IDENT> + <IDENT>', 0)

# after
pattern = parser.compilePattern('<ID> + <ID>', 0)
Defensive patterns

Strategy: validation

Validate before calling

tag_names = {t for t in parser.symbolicNames if t}
assert 'ID' in tag_names, 'unknown token tag'
pattern = parser.compilePattern('<ID>', 0)

Type guard

def is_known_token(tag: str, parser) -> bool:
    return parser.getTokenType(tag) != -1  # Token.INVALID_TYPE

Try / catch

try:
    p = matcher.compileTreePattern(pat, idx)
except Exception as e:
    if str(e).startswith('Unknown token'):
        # unknown token tag: check parser.symbolicNames
        ...

Prevention

When it happens

Trigger: Compiling a pattern containing a tag like <ID>, <STRING>, or <myLabel:TOKEN> where that token name does not exist in the parser's vocabulary. Also triggered when a rule tag is misspelled with a capital first letter (e.g. <Expr> instead of <expr>).

Common situations: Pattern written for a different grammar, renamed tokens after a grammar change, or case confusion between token names (uppercase) and rule names (lowercase).

Related errors

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/c73743d15d2625fc. Report an issue: GitHub.