antlr/antlr4 · error · IllegalStateException

cannot consume EOF

Error message

cannot consume EOF

What it means

UnbufferedTokenStream.consume() first checks LA(1); when LA(1) is Token.EOF, the stream is positioned on the single end-of-input sentinel and there is no next token to consume. ANTLR therefore refuses to advance past EOF with an IllegalStateException. This is a stream-contract violation, not a normal syntax error.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java:139

		return "";
	}


	@Override
	public String getText(RuleContext ctx) {
		return getText(ctx.getSourceInterval());
	}


	@Override
	public String getText(Token start, Token stop) {
		return getText(Interval.of(start.getTokenIndex(), stop.getTokenIndex()));
	}

	@Override
	public void consume() {
		if (LA(1) == Token.EOF) {
			throw new IllegalStateException("cannot consume EOF");
		}

		// buf always has at least tokens[p==0] in this method due to ctor
		lastToken = tokens[p];   // track last token for LT(-1)

		// if we're at last token and no markers, opportunity to flush buffer
		if ( p == n-1 && numMarkers==0 ) {
			n = 0;
			p = -1; // p++ will leave this at 0
			lastTokenBufferStart = lastToken;
		}

		p++;
		currentTokenIndex++;
		sync(1);
	}

	/** Make sure we have 'need' elements from current position {@link #p p}. Last valid

View on GitHub (pinned to 7d5770395b)

Solutions

1. Check tokens.LA(1) != Token.EOF before every consume().
2. Fix custom while/for loops so EOF is the loop terminator and is never consumed.
3. If generated parsing reaches this point, debug the custom error strategy or TokenSource that advances past the sentinel.
4. Verify a custom TokenSource emits exactly one EOF token after all real tokens.

Example fix

// before
while (!done) {
    process(tokens.LT(1));
    tokens.consume(); // throws when LT(1) is EOF
}

// after
while (tokens.LA(1) != Token.EOF) {
    process(tokens.LT(1));
    tokens.consume();
}

if (tokens.LA(1) == Token.EOF) {
    // no real token remains; do not call consume()
}

try {
    tokens.consume();
} catch (IllegalStateException e) {
    if (tokens.LA(1) == Token.EOF) {
        // treat as clean end of input
    } else {
        throw e;
    }
}

When it happens

Trigger: Calling consume() after the stream has reached EOF, including on an empty token source whose first token is EOF; custom loops that consume one token too many; error-recovery or listener code that consumes while recovering at EOF; or a custom TokenSource that returns EOF before the input expected by the parser.

Common situations: Hand-written token-stream drivers, custom parse error strategies, custom TokenSource implementations, and empty input files. Generated parsers normally stop at EOF, so a hit here usually means custom consumption logic or a token source that emits EOF at the wrong place.

Related errors

• cannot consume EOF
• release() called with an invalid marker.
• cannot seek to negative index {index}
• seek to index outside buffer: {index} not in {bufferStartInd
• Unbuffered stream cannot know its size

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/f501750532a69674. Report an issue: GitHub.