antlr/antlr4 · error · ArgumentNullException

tokenSource cannot be null

Error message

tokenSource cannot be null

What it means

BufferedTokenStream's constructor requires an ITokenSource; the stream lazily pulls tokens from that source on demand and cannot function without one. The null check fails fast in the constructor rather than later during Fetch(), where a NullReferenceException would be much harder to trace back to the constructor call.

Source

Thrown at runtime/CSharp/src/BufferedTokenStream.cs:115

        /// and
        /// <see cref="p"/>
        /// instead of calling
        /// <see cref="LA(int)"/>
        /// .</li>
        /// <li>
        /// <see cref="Fetch(int)"/>
        /// : The check to prevent adding multiple EOF symbols into
        /// <see cref="tokens"/>
        /// is trivial with this field.</li>
        /// </ul>
        /// </summary>
        protected internal bool fetchedEOF;

        public BufferedTokenStream(ITokenSource tokenSource)
        {
            if (tokenSource == null)
            {
                throw new ArgumentNullException("tokenSource cannot be null");
            }
            this._tokenSource = tokenSource;
        }

        public virtual ITokenSource TokenSource
        {
            get
            {
                return _tokenSource;
            }
        }

        public virtual int Index
        {
            get
            {
                return p;
            }

View on GitHub (pinned to 7d5770395b)

Solutions

1. Check the token source for null before constructing the stream and throw a descriptive error naming the source
2. Fix the upstream factory so it throws on failure instead of returning null (fail fast, no null propagation)
3. For tests, pass a real generated lexer or an explicit ListTokenSource over sample tokens

Example fix

// before
ITokenSource src = CreateLexerOrNull(path); // returns null on bad path
var stream = new BufferedTokenStream(src); // ArgumentNullException

// after
ITokenSource src = CreateLexer(path) ?? throw new InvalidOperationException($"No lexer for {path}");
var stream = new BufferedTokenStream(src);

if (tokenSource == null)
    throw new InvalidOperationException($"Token source for {sourceName} was not created; check lexer factory/DI registration.");
var stream = new BufferedTokenStream(tokenSource);

static bool IsUsableTokenSource(ITokenSource src) => src != null && src.InputStream != null;

When it happens

Trigger: new BufferedTokenStream(null), typically because a factory method returned null (e.g. a lexer creation function that failed silently) and its result was passed straight through.

Common situations: Dependency-injection containers resolving ITokenSource to null because the lexer wasn't registered; helper methods like CreateLexer(path) returning null on file-not-found instead of throwing; test code passing a mock token source that is null by default.

Related errors

• tokenSource cannot be null
• tokens cannot be null
• tokens cannot be null
• delegates
• target cannot be null.

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/ed03890460a2f808. Report an issue: GitHub.