antlr/antlr4 · error · IllegalStateException

cannot consume EOF

Error message

cannot consume EOF

What it means

CodePointCharStream.consume() throws IllegalStateException when position == size, i.e. the cursor sits on the EOF sentinel and there is nothing left to consume. This is ANTLR's IntStream contract: consume() past the end is a programming error, and LA(1) at that point returns IntStream.EOF. Lexers built on the stream never trigger this; hand-written stream manipulation does.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/CodePointCharStream.java:96

						name,
						codePointBuffer.charArray(),
						codePointBuffer.arrayOffset());
			case INT:
				return new CodePoint32BitCharStream(
						codePointBuffer.position(),
						codePointBuffer.remaining(),
						name,
						codePointBuffer.intArray(),
						codePointBuffer.arrayOffset());
		}
		throw new UnsupportedOperationException("Not reached");
	}

	@Override
	public final void consume() {
		if (size - position == 0) {
			assert LA(1) == IntStream.EOF;
			throw new IllegalStateException("cannot consume EOF");
		}
		position = position + 1;
	}

	@Override
	public final int index() {
		return position;
	}

	@Override
	public final int size() {
		return size;
	}

	/** mark/release do nothing; we have entire buffer */
	@Override
	public final int mark() {
		return -1;

View on GitHub (pinned to 7d5770395b)

Solutions

1. Check LA(1) != IntStream.EOF (or index() < size()) before each consume()
2. Prefer indexed reads (LA(i), index(), size()) over consume() for read-only scanning
3. Audit custom Lexer/TokenSource subclasses for consume() calls not guarded by an EOF check

Example fix

// before
while (true) { process(cs.LA(1)); cs.consume(); } // throws at EOF

// after
while (cs.LA(1) != IntStream.EOF) { process(cs.LA(1)); cs.consume(); }

if (cs.index() < cs.size()) { // equivalently LA(1) != IntStream.EOF
  cs.consume();
}

When it happens

Trigger: Calling charStream.consume() in a custom loop after index() == size() - 1 has already been consumed once more; custom TokenSource implementations advancing a shared CharStream past the end; a Lexer subclass overriding nextToken() incorrectly.

Common situations: Manual scanning experiments over CharStream; porting code from another stream API that permits consuming at end; bugs in lexer wrappers that double-consume the final character.

Related errors

• cannot consume EOF
• cannot consume EOF
• cannot consume EOF
• seek to index outside buffer: {index} not in {bufferStartInd
• the interval extends past the end of the stream

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/fac36d83d27261f5. Report an issue: GitHub.