antlr/antlr4 · error · Exception

Invalid tokens or characters at index %d in path '%s'

Error message

Invalid tokens or characters at index %d in path '%s'

What it means

Raised by XPath.py in the Python3 runtime when the XPathLexer throws LexerNoViableAltException while scanning the XPath string, i.e. the path contains a character the XPath grammar does not accept. Valid XPath input is limited to '/', '//', '!', '*', token/rule names, and quoted strings.

Source

Thrown at runtime/Python3/src/antlr4/xpath/XPath.py:94

        self.parser = parser
        self.path = path
        self.elements = self.split(path)

    def split(self, path:str):
        input = InputStream(path)
        lexer = XPathLexer(input)
        def recover(self, e):
            raise e
        lexer.recover = recover
        lexer.removeErrorListeners()
        lexer.addErrorListener(ErrorListener()) # XPathErrorListener does no more
        tokenStream = CommonTokenStream(lexer)
        try:
            tokenStream.fill()
        except LexerNoViableAltException as e:
            pos = lexer.column
            msg = "Invalid tokens or characters at index %d in path '%s'" % (pos, path)
            raise Exception(msg, e)

        tokens = iter(tokenStream.tokens)
        elements = list()
        for el in tokens:
            invert = False
            anywhere = False
            # Check for path separators, if none assume root
            if el.type in [XPathLexer.ROOT, XPathLexer.ANYWHERE]:
                anywhere = el.type == XPathLexer.ANYWHERE
                next_el = next(tokens, None)
                if not next_el:
                    raise Exception('Missing element after %s' % el.getText())
                else:
                    el = next_el
            # Check for bangs
            if el.type == XPathLexer.BANG:
                invert = True
                next_el = next(tokens, None)

View on GitHub (pinned to 7d5770395b)

Solutions

1. Rewrite the XPath using only supported syntax: /, //, !, *, IDENT, 'string'
2. Remove XML/JSONPath-style predicates and attribute selectors
3. Check the path character-by-character at the reported column index

Example fix

# before
nodes = XPath.findAll(tree, "/stmt[@id=3]", parser)

# after
nodes = XPath.findAll(tree, "/stmt", parser)

import re
if re.search(r"[^A-Za-z0-9_/'!*\" ]", xpath):
    raise ValueError('unsupported XPath syntax')

try:
    nodes = XPath.findAll(tree, path, parser)
except Exception as e:
    if 'Invalid tokens or characters' in str(e):
        # path uses unsupported XPath features; rewrite it
        ...

When it happens

Trigger: Calling XPath(parser, path) or XPath.findAll(tree, path, parser) with a path containing illegal characters such as '..', '@', '[', ']', or ':'.

Common situations: Assuming ANTLR XPath supports standard XML/JSONPath syntax (predicates, '@attr', '..') when it is a much smaller language; pasting a file path or selector by mistake.

Understand the failure class

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• Invalid tokens or characters at index ${pos} in path '${path
• nextToken requires a non-null input stream.
• Empty Stack
• The current recognizer does not provide a list of token name
• The current recognizer does not provide a list of rule names

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/6a3af4da9387a664. Report an issue: GitHub.