{"record":{"id":"6eeb2bf1f97c40c9","repo":"antlr/antlr4","slug":"cannot-consume-eof-6eeb2b","errorCode":null,"errorMessage":"cannot consume EOF","messagePattern":"cannot consume EOF","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"runtime/Python3/src/antlr4/BufferedTokenStream.py","lineNumber":98,"sourceCode":"        self.lazyInit()\n        return self.tokens[index]\n\n    def consume(self):\n        skipEofCheck = False\n        if self.index >= 0:\n            if self.fetchedEOF:\n                # the last token in tokens is EOF. skip check if p indexes any\n                # fetched token except the last.\n                skipEofCheck = self.index < len(self.tokens) - 1\n            else:\n               # no EOF token in tokens. skip check if p indexes a fetched token.\n                skipEofCheck = self.index < len(self.tokens)\n        else:\n            # not yet initialized\n            skipEofCheck = False\n\n        if not skipEofCheck and self.LA(1) == Token.EOF:\n            raise IllegalStateException(\"cannot consume EOF\")\n\n        if self.sync(self.index + 1):\n            self.index = self.adjustSeekIndex(self.index + 1)\n\n    # Make sure index {@code i} in tokens has a token.\n    #\n    # @return {@code true} if a token is located at index {@code i}, otherwise\n    #    {@code false}.\n    # @see #get(int i)\n    #/\n    def sync(self, i:int):\n        n = i - len(self.tokens) + 1 # how many more elements we need?\n        if n > 0 :\n            fetched = self.fetch(n)\n            return fetched >= n\n        return True\n\n    # Add {@code n} elements to buffer.","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Python3/src/antlr4/BufferedTokenStream.py#L80-L116","documentation":"BufferedTokenStream.consume() raises IllegalStateException when the next token (LA(1)) is EOF and the skip-check does not apply — i.e., the current index already sits on the final EOF token. Consuming EOF is a logic error: there is nothing after end of input, and the stream guard prevents silent index corruption.","triggerScenarios":"Calling consume() after the stream has already advanced onto the EOF token (index == len(tokens)-1 with fetchedEOF); a hand-written parser loop that consumes without checking for EOF; custom token-stream wrappers that call consume unconditionally in a while loop.","commonSituations":"Iterating tokens with 'while True: tokens.consume()' instead of 'while tokens.LA(1) != Token.EOF'; grammar/lexer issues (unterminated string or comment) that make the parser request more tokens after EOF; custom code ported from a stream without the EOF guard.","solutions":["Loop condition must test LA(1): consume only while tokens.LA(1) != Token.EOF","If a parser drives the stream, check the grammar handles end-of-input (every loop rule has an EOF exit) so the parser never demands a token past EOF","For manual iteration, use the token count: iterate while index < len via the buffered API rather than consuming blindly"],"exampleFix":"# before\nwhile True:\n    tok = tokens.LT(1)\n    process(tok)\n    tokens.consume()  # throws once EOF is the current token\n\n# after\nwhile tokens.LA(1) != Token.EOF:\n    tok = tokens.LT(1)\n    process(tok)\n    tokens.consume()","handlingStrategy":"validation","validationCode":"# Python: guard consume with the EOF check\nfrom antlr4.Token import Token\nif tokens.LA(1) != Token.EOF:\n    tokens.consume()","typeGuard":null,"tryCatchPattern":"from antlr4.error.Errors import IllegalStateException\ntry:\n    tokens.consume()\nexcept IllegalStateException as ex:\n    if \"cannot consume EOF\" in str(ex):\n        pass  # already at end: stop the token loop\n    else:\n        raise","preventionTips":["Make 'while tokens.LA(1) != Token.EOF:' the standard shape of every manual token loop","After lexer errors, re-check LA(1) rather than assuming a fixed number of remaining tokens","Wrap third-party token-walking code with an EOF assert before it consumes"],"tags":["antlr","python","token-stream","eof","consume"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}