antlr/antlr4 · error · ArgumentException

Invalid tokens or characters at index ${pos} in path '${path

Error message

Invalid tokens or characters at index ${pos} in path '${path}'

What it means

XPath.Split lexes the XPath string with XPathLexer; when the lexer cannot match any rule at the current position it throws LexerNoViableAltException, which Split converts into ArgumentException naming the column of the offending character. In other words, the XPath query contains characters or tokens that the XPath grammar does not support.

Source

Thrown at runtime/CSharp/src/Tree/Xpath/XPath.cs:114

                @in = new AntlrInputStream(new StringReader(path));
            }
            catch (IOException ioe)
            {
                throw new ArgumentException("Could not read path: " + path, ioe);
            }
            XPathLexer lexer = new _XPathLexer_87(@in);
            lexer.RemoveErrorListeners();
            lexer.AddErrorListener(new XPathLexerErrorListener());
            CommonTokenStream tokenStream = new CommonTokenStream(lexer);
            try
            {
                tokenStream.Fill();
            }
            catch (LexerNoViableAltException e)
            {
                int pos = lexer.Column;
                string msg = "Invalid tokens or characters at index " + pos + " in path '" + path + "'";
                throw new ArgumentException(msg, e);
            }
            IList<IToken> tokens = tokenStream.GetTokens();
            //		System.out.println("path="+path+"=>"+tokens);
            IList<XPathElement> elements = new List<XPathElement>();
            int n = tokens.Count;
            int i = 0;
            while (i < n)
            {
                IToken el = tokens[i];
                IToken next = null;
                switch (el.Type)
                {
                    case XPathLexer.Root:
                    case XPathLexer.Anywhere:
                    {
                        bool anywhere = el.Type == XPathLexer.Anywhere;
                        i++;
                        next = tokens[i];

View on GitHub (pinned to 7d5770395b)

Solutions

Rewrite the query using the supported subset: '/', '//', '*', identifier, and '!' anywhere (e.g. "//funcDecl", "/translationUnit/decl/*").
Remove unsupported constructs (predicates, attributes, parent axes) and do the fine-grained filtering in C# over the returned nodes.
Verify every name in the path against parser.RuleNames / Vocabulary so the elements resolve after lexing.
Catch ArgumentException around XPath construction to report the bad query with its index when queries come from users.

Example fix

// before
var xp = new XPath(parser, "//funcDecl[@name='foo']"); // predicates unsupported

// after
var xp = new XPath(parser, "//funcDecl");
foreach (var ctx in xp.Evaluate(tree).OfType<FuncDeclContext>())
    if (ctx.name?.GetText() == "foo") { /* ... */ }

Defensive patterns

Strategy: try-catch

Try / catch

try { var xp = new XPath(parser, path); } catch (ArgumentException ex) when (ex.Message.Contains("Invalid tokens or characters")) { /* surface 'unsupported XPath syntax', suggest the ANTLR subset: / // * word ! */ }

Prevention

Learn the supported XPath subset: '/', '//', '*', identifiers, '!'; no predicates, attributes, or W3C axes.
Do fine-grained node filtering in C# over Evaluate() results instead of encoding it in the path.
Validate user-supplied queries against a whitelist of allowed characters before constructing XPath.

When it happens

Trigger: new XPath(parser, "//declaration/*bad") — invalid characters, unsupported operators, stray quotes, or whitespace the XPath lexer rejects. Valid XPath here is the ANTLR parse-tree subset: '/', '//', '*', word, '!', and rule-or-token names.

Common situations: Assuming W3C XPath syntax (predicates like [1], '@attr', '..') works — ANTLR's XPath is a tiny subset; copy-pasting real XPath expressions from elsewhere; grammar-specific names not matching the parser's rule/token names.

Understand the failure class

Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/04ca098a74f7614d. Report an issue: GitHub.