antlr/antlr4 · error · IllegalStateException

nextToken requires a non-null input stream.

Error message

nextToken requires a non-null input stream.

What it means

Lexer.nextToken() throws IllegalStateException when the lexer's _input CharStream is null — matching requires reading characters, so there is nothing sensible to return. The lexer normally receives its CharStream via setInputStream() or the generated constructor; a null input means the lexer was constructed or reset incorrectly (e.g. reused after being passed null).

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/Lexer.java:114

		_tokenStartCharIndex = -1;
		_tokenStartCharPositionInLine = -1;
		_tokenStartLine = -1;
		_text = null;

		_hitEOF = false;
		_mode = Lexer.DEFAULT_MODE;
		_modeStack.clear();

		getInterpreter().reset();
	}

	/** Return a token from this source; i.e., match a token on the char
	 *  stream.
	 */
	@Override
	public Token nextToken() {
		if (_input == null) {
			throw new IllegalStateException("nextToken requires a non-null input stream.");
		}

		// Mark start location in char stream so unbuffered streams are
		// guaranteed at least have text of current token
		int tokenStartMarker = _input.mark();
		try{
			outer:
			while (true) {
				if (_hitEOF) {
					emitEOF();
					return _token;
				}

				_token = null;
				_channel = Token.DEFAULT_CHANNEL;
				_tokenStartCharIndex = _input.index();
				_tokenStartCharPositionInLine = getInterpreter().getCharPositionInLine();
				_tokenStartLine = getInterpreter().getLine();

View on GitHub (pinned to 7d5770395b)

Solutions

1. Always set the input before tokenizing: lexer.setInputStream(CharStreams.fromString(input)) or construct with the stream
2. Create a fresh Lexer per input (recommended; they are cheap) rather than reusing across files
3. Null-check the CharStream-producing code path (file read, URL fetch) before assigning it to the lexer

Example fix

// before
MyLexer lexer = new MyLexer(null); // later nextToken() throws

// after
MyLexer lexer = new MyLexer(CharStreams.fromString(input));
// or: lexer.setInputStream(CharStreams.fromPath(path));

CharStream input = CharStreams.fromString(text);
if (input != null) {
  lexer.setInputStream(input);
}

When it happens

Trigger: Constructing a generated Lexer with the no-arg (or null) constructor and forgetting setInputStream(CharStream); calling reset() then nextToken() on a lexer whose input was never set; DI frameworks instantiating the lexer without wiring the stream.

Common situations: Reusing a single Lexer object across files and forgetting setInputStream() between parses; testing harnesses that new the lexer directly; refactoring that removed the constructor argument.

Related errors

• nextToken requires a non-null input stream.
• Parser can't discover a lexer to use
• The specified lexer action type %s is not valid.
• The specified lexer action type %s is not valid.
• Precedence predicates are not supported in lexers.

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/8ba7643eeb403c9e. Report an issue: GitHub.