antlr/antlr4 · error · InvalidOperationException

nextToken requires a non-null input stream.

Error message

nextToken requires a non-null input stream.

What it means

Lexer.NextToken() pulls characters from the ICharStream stored in _input (set by the Recognizer base's InputStream setter / constructor). If the lexer has no input stream attached, there is nothing to tokenize, so the method throws InvalidOperationException immediately rather than NRE-ing inside the matching loop. The lexer also resets per-token state (tokenStartCharIndex, mode stack) from _input, so the check is essential.

Source

Thrown at runtime/CSharp/src/Lexer.cs:145

            _hitEOF = false;
            _mode = Antlr4.Runtime.Lexer.DEFAULT_MODE;
            _modeStack.Clear();
            Interpreter.Reset();
        }

        /// <summary>
        /// Return a token from this source; i.e., match a token on the char
        /// stream.
        /// </summary>
        /// <remarks>
        /// Return a token from this source; i.e., match a token on the char
        /// stream.
        /// </remarks>
        public virtual IToken NextToken()
        {
            if (_input == null)
            {
                throw new InvalidOperationException("nextToken requires a non-null input stream.");
            }
            // Mark start location in char stream so unbuffered streams are
            // guaranteed at least have text of current token
            int tokenStartMarker = _input.Mark();
            try
            {
                while (true)
                {
                    if (_hitEOF)
                    {
                        EmitEOF();
                        return _token;
                    }
                    _token = null;
                    _channel = TokenConstants.DefaultChannel;
                    _tokenStartCharIndex = _input.Index;
                    _tokenStartColumn = Interpreter.Column;
                    _tokenStartLine = Interpreter.Line;

View on GitHub (pinned to 7d5770395b)

Solutions

1. Always construct the lexer with the input stream: new MyLexer(new AntlrInputStream(text))
2. If you must set it later, assign lexer.InputStream before the first NextToken() call
3. For pooled/reused lexers, re-assign the stream on every checkout and assert it is non-null before use

Example fix

// before
var lexer = new MyLexer(); // parameterless, no stream
lexer.Reset();
IToken t = lexer.NextToken(); // InvalidOperationException

// after
var lexer = new MyLexer(new AntlrInputStream(sourceText));
IToken t = lexer.NextToken();

if (lexer.InputStream == null)
    throw new InvalidOperationException("Lexer has no input stream; assign InputStream before lexing.");
IToken t = lexer.NextToken();

static bool IsReadyToLex(Lexer lx) => lx != null && lx.InputStream != null;

When it happens

Trigger: Calling NextToken() on a lexer constructed without a stream, or after setting InputStream to null; a lexer subclass whose constructor forgets to pass the stream to base; DI frameworks creating the lexer with a parameterless constructor and never assigning InputStream.

Common situations: Using the parameterless constructor for test harnesses then forgetting lexer.InputStream = new AntlrInputStream(text); resetting a pooled lexer instance between requests and clearing InputStream; subclass constructors that omit the : base(input) call.

Related errors

• The specified lexer action type {0} is not valid.
• Precedence predicates are not supported in lexers.
• target cannot be null.
• tokenSource cannot be null
• The ATN must be a lexer ATN.

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/8742ecfe9eed72a3. Report an issue: GitHub.