antlr/antlr4 · error · ArgumentNullException

tokens cannot be null

Error message

tokens cannot be null

What it means

ListTokenSource replays a fixed list of already-created tokens (typically tokens captured from a previous lexing pass, e.g. during error recovery or re-lexing). The constructor requires the list to be non-null; an empty list is fine (it yields EOF), but null has no meaningful interpretation for a source whose whole purpose is that list.

Source

Thrown at runtime/CSharp/src/ListTokenSource.cs:137

        /// <see cref="SourceName()"/>
        /// will attempt to infer the name from
        /// the next
        /// <see cref="IToken"/>
        /// (or the previous token if the end of the input has
        /// been reached).
        /// </param>
        /// <exception>
        /// NullPointerException
        /// if
        /// <paramref name="tokens"/>
        /// is
        /// <see langword="null"/>
        /// </exception>
        public ListTokenSource(IList<IToken> tokens, string sourceName)
        {
            if (tokens == null)
            {
                throw new ArgumentNullException("tokens cannot be null");
            }
            this.tokens = tokens;
            this.sourceName = sourceName;
        }

        /// <summary><inheritDoc/></summary>
        public virtual int Column
        {
            get
            {
                if (i < tokens.Count)
                {
                    return tokens[i].Column;
                }
                else
                {
                    if (eofToken != null)
                    {

View on GitHub (pinned to 7d5770395b)

Solutions

1. Pass a non-null list; use an empty list when no tokens were captured: tokens ?? new List<IToken>()
2. Make upstream token-capture code return an empty list rather than null on failure
3. For tests, build a small list of CommonToken objects explicitly

Example fix

// before
IList<IToken> captured = MaybeCaptureTokens(input); // null when capture fails
var src = new ListTokenSource(captured, "replay"); // ArgumentNullException

// after
IList<IToken> captured = MaybeCaptureTokens(input) ?? new List<IToken>();
var src = new ListTokenSource(captured, "replay");

var source = new ListTokenSource(tokens ?? new List<IToken>(), sourceName);

static bool HasTokenList(IList<IToken> t) => t != null;

When it happens

Trigger: new ListTokenSource(null) or new ListTokenSource(tokens: null, sourceName: ...) — usually a variable that was never assigned, a filter that returned null instead of an empty list, or a capture step that was skipped.

Common situations: Re-lexing pipelines where the first pass failed and produced no token list; LINQ pipelines ending in FirstOrDefault() instead of ToList(); unit tests that stub the token producer as null.

Related errors

• tokens cannot be null
• tokenSource cannot be null
• tokenSource cannot be null
• delegates
• target cannot be null.

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/acbaac3a43a120a6. Report an issue: GitHub.