{"record":{"id":"e795077c74732d98","repo":"antlr/antlr4","slug":"cannot-consume-eof-e79507","errorCode":null,"errorMessage":"cannot consume EOF","messagePattern":"cannot consume EOF","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"runtime/Python3/src/antlr4/InputStream.py","lineNumber":45,"sourceCode":"    @property\n    def index(self):\n        return self._index\n\n    @property\n    def size(self):\n        return self._size\n\n    # Reset the stream so that it's in the same state it was\n    #  when the object was created *except* the data array is not\n    #  touched.\n    #\n    def reset(self):\n        self._index = 0\n\n    def consume(self):\n        if self._index >= self._size:\n            assert self.LA(1) == Token.EOF\n            raise Exception(\"cannot consume EOF\")\n        self._index += 1\n\n    def LA(self, offset: int):\n        if offset==0:\n            return 0 # undefined\n        if offset<0:\n            offset += 1 # e.g., translate LA(-1) to use offset=0\n        pos = self._index + offset - 1\n        if pos < 0 or pos >= self._size: # invalid\n            return Token.EOF\n        return self.data[pos]\n\n    def LT(self, offset: int):\n        return self.LA(offset)\n\n    # mark/release do nothing; we have entire buffer\n    def mark(self):\n        return -1","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Python3/src/antlr4/InputStream.py#L27-L63","documentation":"InputStream.consume() raises Exception('cannot consume EOF') when _index >= _size, i.e., the character cursor is already at or past the last element of the character data. This guards the raw char stream (not tokens) against reads past end of input; the preceding assert also expects LA(1) to be EOF at that point.","triggerScenarios":"Custom code calling inputStream.consume() in a loop without checking LA(1) == Token.EOF; a hand-written lexer or scanner stepping past the final character; consuming after reset() misuse or on an empty InputStream where _size == 0 and the very first consume() fails.","commonSituations":"Building custom lexers on top of antlr4.InputStream; porting C-style loops ('while not eof: consume()') where the EOF check is missing or inverted; empty input files hitting the immediate _index >= _size case.","solutions":["Check self.LA(1) != Token.EOF (equivalently _index < _size) before every consume()","For empty inputs, short-circuit before entering the consume loop","Prefer letting the generated Lexer drive the InputStream instead of consuming it manually"],"exampleFix":"# before\nwhile True:\n    ch = stream.LA(1)\n    handle(ch)\n    stream.consume()  # throws at end of data\n\n# after\nwhile stream.LA(1) != Token.EOF:\n    ch = stream.LA(1)\n    handle(ch)\n    stream.consume()","handlingStrategy":"validation","validationCode":"# Python: check the character cursor before consuming\nfrom antlr4.Token import Token\nif stream.LA(1) != Token.EOF:\n    stream.consume()","typeGuard":null,"tryCatchPattern":"try:\n    stream.consume()\nexcept Exception as ex:\n    if \"cannot consume EOF\" in str(ex):\n        pass  # end of character data reached: exit the scan loop\n    else:\n        raise","preventionTips":["Treat LA(1) == Token.EOF as the only loop condition for custom scanners","Handle empty input (size 0) before entering any consume loop","Let the generated Lexer own the InputStream instead of consuming it by hand where possible"],"tags":["antlr","python","char-stream","eof","consume"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}