antlr/antlr4 · error · IllegalStateException

cannot consume EOF

Error message

cannot consume EOF

What it means

UnbufferedCharStream.consume() throws IllegalStateException when LA(1) is EOF, because consuming past the end of the character input is a protocol violation for ANTLR streams. Unlike buffered streams, the unbuffered stream keeps only a small window, so it checks the current lookahead symbol before advancing.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java:123

		this(input, bufferSize, StandardCharsets.UTF_8);
	}

	public UnbufferedCharStream(InputStream input, int bufferSize, Charset charset) {
		this(bufferSize);
		this.input = new InputStreamReader(input, charset);
		fill(1); // prime
	}

	public UnbufferedCharStream(Reader input, int bufferSize) {
		this(bufferSize);
		this.input = input;
		fill(1); // prime
	}

	@Override
	public void consume() {
		if (LA(1) == IntStream.EOF) {
			throw new IllegalStateException("cannot consume EOF");
		}

		// buf always has at least data[p==0] in this method due to ctor
		lastChar = data[p];   // track last char for LA(-1)

		if (p == n-1 && numMarkers==0) {
			n = 0;
			p = -1; // p++ will leave this at 0
			lastCharBufferStart = lastChar;
		}

		p++;
		currentCharIndex++;
		sync(1);
	}

	/**
	 * Make sure we have 'need' elements from current position {@link #p p}.

View on GitHub (pinned to 7d5770395b)

Solutions

1. Check LA(1) != IntStream.EOF before every consume()
2. In custom lexer overrides, follow the generated nextToken() pattern which stops at EOF
3. If you need unrestricted lookbehind/re-reads, use ANTLRInputStream (buffered) instead of UnbufferedCharStream

Example fix

// before
while (true) { stream.consume(); } // throws at EOF

// after
while (stream.LA(1) != IntStream.EOF) {
    processChar(stream.LA(1));
    stream.consume();
}

if (stream.LA(1) != IntStream.EOF) {
    stream.consume();
}

try { stream.consume(); } catch (IllegalStateException e) { /* at EOF: stop scanning */ }

When it happens

Trigger: Calling stream.consume() when LA(1) already returned IntStream.EOF (-1); a hand-written scanner loop that consumes without checking for EOF; a Lexer subclass whose nextToken() logic consumes after EOF was reached.

Common situations: Custom lexer code or manual CharStream iteration on large files where UnbufferedCharStream was chosen to save memory; mismatch between the generated lexer's EOF handling and added custom consume() calls.

Related errors

• cannot consume EOF
• cannot consume EOF
• cannot consume EOF
• seek to index outside buffer: {index} not in {bufferStartInd
• the interval extends past the end of the stream

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/fa2f9d58d4715aff. Report an issue: GitHub.