What causes "Parser can't discover a lexer to use" in antlr/antlr4?

Pass the lexer explicitly: parser.compileParseTreePattern(pattern, ruleIndex, MyLexer(None))

antlr/antlr4 · error · UnsupportedOperationException

Parser can't discover a lexer to use

Error message

Parser can't discover a lexer to use

What it means

Parser.compileParseTreePattern needs a Lexer to tokenize the pattern string (e.g., '<ID> + 0'). If the explicit lexer argument is None, it tries to discover one from the current token stream's tokenSource; when that source is not a Lexer instance (typical for ListTokenSource or a custom TokenSource), it raises UnsupportedOperationException('Parser can\'t discover a lexer to use').

Source

Thrown at runtime/Python3/src/antlr4/Parser.py:288

    # The preferred method of getting a tree pattern. For example, here's a
    # sample use:
    #
    # <pre>
    # ParseTree t = parser.expr();
    # ParseTreePattern p = parser.compileParseTreePattern("&lt;ID&gt;+0", MyParser.RULE_expr);
    # ParseTreeMatch m = p.match(t);
    # String id = m.get("ID");
    # </pre>
    #
    def compileParseTreePattern(self, pattern:str, patternRuleIndex:int, lexer:Lexer = None):
        if lexer is None:
            if self.getTokenStream() is not None:
                tokenSource = self.getTokenStream().tokenSource
                if isinstance( tokenSource, Lexer ):
                    lexer = tokenSource
        if lexer is None:
            raise UnsupportedOperationException("Parser can't discover a lexer to use")

        m = ParseTreePatternMatcher(lexer, self)
        return m.compile(pattern, patternRuleIndex)


    def getInputStream(self):
        return self.getTokenStream()

    def setInputStream(self, input:InputStream):
        self.setTokenStream(input)

    def getTokenStream(self):
        return self._input

    # Set the token stream and reset the parser.#
    def setTokenStream(self, input:TokenStream):
        self._input = None
        self.reset()

View on GitHub (pinned to 7d5770395b)

Solutions

Pass the lexer explicitly: parser.compileParseTreePattern(pattern, ruleIndex, MyLexer(None))
Keep the original lexer instance around when you switch the parser to a list-based token source and hand it to every pattern call
Alternatively build the parser on a CommonTokenStream over the real lexer so discovery succeeds

Example fix

# before
parser = MyParser(CommonTokenStream(ListTokenSource(recorded_tokens)))
p = parser.compileParseTreePattern("<ID>", MyParser.RULE_expr)  # no lexer to find

# after
lexer = MyLexer(None)
p = parser.compileParseTreePattern("<ID>", MyParser.RULE_expr, lexer)

Defensive patterns

Strategy: validation

Validate before calling

# Python: pass a lexer explicitly whenever the stream's source may not be one
lexer = lexer or MyLexer(None)
pattern = parser.compileParseTreePattern(pattern_str, rule_index, lexer)

Type guard

# Python
from antlr4.Lexer import Lexer
def token_source_is_lexer(parser) -> bool:
    src = parser.getTokenStream().tokenSource if parser.getTokenStream() else None
    return isinstance(src, Lexer)

Try / catch

try:
    p = parser.compileParseTreePattern(pat, idx)
except UnsupportedOperationException as ex:
    if "discover a lexer" in str(ex):
        p = parser.compileParseTreePattern(pat, idx, MyLexer(None))
    else:
        raise

Prevention

Store the lexer used for the parse and pass it to every pattern-compile call
When replaying tokens from a list, keep the original lexer instance alongside
Treat the implicit lexer discovery as a convenience only — make it explicit in production code

When it happens

Trigger: Calling compileParseTreePattern(pattern, ruleIndex) without the third argument while the parser's input stream is fed by a ListTokenSource or other non-Lexer token source; constructing the parser from a pre-lexed token list for replay/testing; tokenSource wrapped in an adapter object.

Common situations: Test harnesses that replay recorded tokens; IDE/language-server code that parses from cached token lists; refactoring from a lexer-fed stream to a token-list stream while keeping tree-pattern code.

Related errors

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/0c9d3cf8822aa4b6. Report an issue: GitHub.