antlr/antlr4 · error · IllegalStateException

cannot consume EOF

Error message

cannot consume EOF

What it means

BufferedTokenStream.consume() throws IllegalStateException when the stream's lookahead is already at EOF, because consuming the virtual end-of-file token would move the index past the end of the token buffer and corrupt stream invariants. The check is an optimization that inspects fetchedEOF and p instead of calling LA(1). Well-formed parsers never consume EOF, so this exception signals a bug in custom stream usage or hand-written parsing logic.

Source

Thrown at runtime/Java/src/org/antlr/v4/runtime/BufferedTokenStream.java:133

		boolean skipEofCheck;
		if (p >= 0) {
			if (fetchedEOF) {
				// the last token in tokens is EOF. skip check if p indexes any
				// fetched token except the last.
				skipEofCheck = p < tokens.size() - 1;
			}
			else {
				// no EOF token in tokens. skip check if p indexes a fetched token.
				skipEofCheck = p < tokens.size();
			}
		}
		else {
			// not yet initialized
			skipEofCheck = false;
		}

		if (!skipEofCheck && LA(1) == EOF) {
			throw new IllegalStateException("cannot consume EOF");
		}

		if (sync(p + 1)) {
			p = adjustSeekIndex(p + 1);
		}
    }

    /** Make sure index {@code i} in tokens has a token.
	 *
	 * @return {@code true} if a token is located at index {@code i}, otherwise
	 *    {@code false}.
	 * @see #get(int i)
	 */
    protected boolean sync(int i) {
		assert i >= 0;
        int n = i - tokens.size() + 1; // how many more elements we need?
        //System.out.println("sync("+i+") needs "+n);
        if ( n > 0 ) {

View on GitHub (pinned to 7d5770395b)

Solutions

1. Guard consumption: only call consume() when LA(1) != Token.EOF
2. Iterate with get(i) / size() instead of consume() when you just need to scan all tokens
3. Audit custom Parser/TokenStream subclasses for consume() calls made after EOF was matched

Example fix

// before
while (true) { process(tokens.LT(1)); tokens.consume(); } // throws at EOF

// after
while (tokens.LA(1) != Token.EOF) { process(tokens.LT(1)); tokens.consume(); }

if (tokens.LA(1) != Token.EOF) {
  tokens.consume();
}

When it happens

Trigger: Calling tokens.consume() in a manual loop until (and including) the EOF token; custom TokenStream wrappers or parser extensions that call consume() after match(Token.EOF); using an index already at size()-1 (the EOF position) and consuming again.

Common situations: Hand-rolled token iteration that uses while (t.getType() != Token.EOF) { ... consume(); } with an off-by-one; adapting generated parser harnesses; writing a custom syntax highlighter over the token stream.

Related errors

• cannot consume EOF
• token index {} out of range 0..{}
• cannot consume EOF
• cannot consume EOF
• cannot consume EOF

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/4d96ea68ab094519. Report an issue: GitHub.