antlr/antlr4 · error · Exception

cannot consume EOF

Error message

cannot consume EOF

What it means

InputStream.consume() raises Exception('cannot consume EOF') when _index >= _size, i.e., the character cursor is already at or past the last element of the character data. This guards the raw char stream (not tokens) against reads past end of input; the preceding assert also expects LA(1) to be EOF at that point.

Source

Thrown at runtime/Python3/src/antlr4/InputStream.py:45

    @property
    def index(self):
        return self._index

    @property
    def size(self):
        return self._size

    # Reset the stream so that it's in the same state it was
    #  when the object was created *except* the data array is not
    #  touched.
    #
    def reset(self):
        self._index = 0

    def consume(self):
        if self._index >= self._size:
            assert self.LA(1) == Token.EOF
            raise Exception("cannot consume EOF")
        self._index += 1

    def LA(self, offset: int):
        if offset==0:
            return 0 # undefined
        if offset<0:
            offset += 1 # e.g., translate LA(-1) to use offset=0
        pos = self._index + offset - 1
        if pos < 0 or pos >= self._size: # invalid
            return Token.EOF
        return self.data[pos]

    def LT(self, offset: int):
        return self.LA(offset)

    # mark/release do nothing; we have entire buffer
    def mark(self):
        return -1

View on GitHub (pinned to 7d5770395b)

Solutions

1. Check self.LA(1) != Token.EOF (equivalently _index < _size) before every consume()
2. For empty inputs, short-circuit before entering the consume loop
3. Prefer letting the generated Lexer drive the InputStream instead of consuming it manually

Example fix

# before
while True:
    ch = stream.LA(1)
    handle(ch)
    stream.consume()  # throws at end of data

# after
while stream.LA(1) != Token.EOF:
    ch = stream.LA(1)
    handle(ch)
    stream.consume()

# Python: check the character cursor before consuming
from antlr4.Token import Token
if stream.LA(1) != Token.EOF:
    stream.consume()

try:
    stream.consume()
except Exception as ex:
    if "cannot consume EOF" in str(ex):
        pass  # end of character data reached: exit the scan loop
    else:
        raise

When it happens

Trigger: Custom code calling inputStream.consume() in a loop without checking LA(1) == Token.EOF; a hand-written lexer or scanner stepping past the final character; consuming after reset() misuse or on an empty InputStream where _size == 0 and the very first consume() fails.

Common situations: Building custom lexers on top of antlr4.InputStream; porting C-style loops ('while not eof: consume()') where the EOF check is missing or inverted; empty input files hitting the immediate _index >= _size case.

Related errors

• cannot consume EOF
• cannot consume EOF
• cannot consume EOF
• cannot consume EOF
• seek to index outside buffer: {index} not in {bufferStartInd

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/e795077c74732d98. Report an issue: GitHub.