antlr/antlr4 · error · IllegalStateException

cannot consume EOF

Error message

cannot consume EOF

What it means

BufferedTokenStream.consume() raises IllegalStateException when the next token (LA(1)) is EOF and the skip-check does not apply — i.e., the current index already sits on the final EOF token. Consuming EOF is a logic error: there is nothing after end of input, and the stream guard prevents silent index corruption.

Source

Thrown at runtime/Python3/src/antlr4/BufferedTokenStream.py:98

        self.lazyInit()
        return self.tokens[index]

    def consume(self):
        skipEofCheck = False
        if self.index >= 0:
            if self.fetchedEOF:
                # the last token in tokens is EOF. skip check if p indexes any
                # fetched token except the last.
                skipEofCheck = self.index < len(self.tokens) - 1
            else:
               # no EOF token in tokens. skip check if p indexes a fetched token.
                skipEofCheck = self.index < len(self.tokens)
        else:
            # not yet initialized
            skipEofCheck = False

        if not skipEofCheck and self.LA(1) == Token.EOF:
            raise IllegalStateException("cannot consume EOF")

        if self.sync(self.index + 1):
            self.index = self.adjustSeekIndex(self.index + 1)

    # Make sure index {@code i} in tokens has a token.
    #
    # @return {@code true} if a token is located at index {@code i}, otherwise
    #    {@code false}.
    # @see #get(int i)
    #/
    def sync(self, i:int):
        n = i - len(self.tokens) + 1 # how many more elements we need?
        if n > 0 :
            fetched = self.fetch(n)
            return fetched >= n
        return True

    # Add {@code n} elements to buffer.

View on GitHub (pinned to 7d5770395b)

Solutions

1. Loop condition must test LA(1): consume only while tokens.LA(1) != Token.EOF
2. If a parser drives the stream, check the grammar handles end-of-input (every loop rule has an EOF exit) so the parser never demands a token past EOF
3. For manual iteration, use the token count: iterate while index < len via the buffered API rather than consuming blindly

Example fix

# before
while True:
    tok = tokens.LT(1)
    process(tok)
    tokens.consume()  # throws once EOF is the current token

# after
while tokens.LA(1) != Token.EOF:
    tok = tokens.LT(1)
    process(tok)
    tokens.consume()

# Python: guard consume with the EOF check
from antlr4.Token import Token
if tokens.LA(1) != Token.EOF:
    tokens.consume()

from antlr4.error.Errors import IllegalStateException
try:
    tokens.consume()
except IllegalStateException as ex:
    if "cannot consume EOF" in str(ex):
        pass  # already at end: stop the token loop
    else:
        raise

When it happens

Trigger: Calling consume() after the stream has already advanced onto the EOF token (index == len(tokens)-1 with fetchedEOF); a hand-written parser loop that consumes without checking for EOF; custom token-stream wrappers that call consume unconditionally in a while loop.

Common situations: Iterating tokens with 'while True: tokens.consume()' instead of 'while tokens.LA(1) != Token.EOF'; grammar/lexer issues (unterminated string or comment) that make the parser request more tokens after EOF; custom code ported from a stream without the EOF guard.

Related errors

• cannot consume EOF
• cannot consume EOF
• cannot consume EOF
• cannot consume EOF
• cannot consume EOF

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/6eeb2bf1f97c40c9. Report an issue: GitHub.