{"record":{"id":"4d96ea68ab094519","repo":"antlr/antlr4","slug":"cannot-consume-eof-4d96ea","errorCode":null,"errorMessage":"cannot consume EOF","messagePattern":"cannot consume EOF","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"runtime/Java/src/org/antlr/v4/runtime/BufferedTokenStream.java","lineNumber":133,"sourceCode":"\t\tboolean skipEofCheck;\n\t\tif (p >= 0) {\n\t\t\tif (fetchedEOF) {\n\t\t\t\t// the last token in tokens is EOF. skip check if p indexes any\n\t\t\t\t// fetched token except the last.\n\t\t\t\tskipEofCheck = p < tokens.size() - 1;\n\t\t\t}\n\t\t\telse {\n\t\t\t\t// no EOF token in tokens. skip check if p indexes a fetched token.\n\t\t\t\tskipEofCheck = p < tokens.size();\n\t\t\t}\n\t\t}\n\t\telse {\n\t\t\t// not yet initialized\n\t\t\tskipEofCheck = false;\n\t\t}\n\n\t\tif (!skipEofCheck && LA(1) == EOF) {\n\t\t\tthrow new IllegalStateException(\"cannot consume EOF\");\n\t\t}\n\n\t\tif (sync(p + 1)) {\n\t\t\tp = adjustSeekIndex(p + 1);\n\t\t}\n    }\n\n    /** Make sure index {@code i} in tokens has a token.\n\t *\n\t * @return {@code true} if a token is located at index {@code i}, otherwise\n\t *    {@code false}.\n\t * @see #get(int i)\n\t */\n    protected boolean sync(int i) {\n\t\tassert i >= 0;\n        int n = i - tokens.size() + 1; // how many more elements we need?\n        //System.out.println(\"sync(\"+i+\") needs \"+n);\n        if ( n > 0 ) {","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/BufferedTokenStream.java#L115-L151","documentation":"BufferedTokenStream.consume() throws IllegalStateException when the stream's lookahead is already at EOF, because consuming the virtual end-of-file token would move the index past the end of the token buffer and corrupt stream invariants. The check is an optimization that inspects fetchedEOF and p instead of calling LA(1). Well-formed parsers never consume EOF, so this exception signals a bug in custom stream usage or hand-written parsing logic.","triggerScenarios":"Calling tokens.consume() in a manual loop until (and including) the EOF token; custom TokenStream wrappers or parser extensions that call consume() after match(Token.EOF); using an index already at size()-1 (the EOF position) and consuming again.","commonSituations":"Hand-rolled token iteration that uses while (t.getType() != Token.EOF) { ... consume(); } with an off-by-one; adapting generated parser harnesses; writing a custom syntax highlighter over the token stream.","solutions":["Guard consumption: only call consume() when LA(1) != Token.EOF","Iterate with get(i) / size() instead of consume() when you just need to scan all tokens","Audit custom Parser/TokenStream subclasses for consume() calls made after EOF was matched"],"exampleFix":"// before\nwhile (true) { process(tokens.LT(1)); tokens.consume(); } // throws at EOF\n\n// after\nwhile (tokens.LA(1) != Token.EOF) { process(tokens.LT(1)); tokens.consume(); }","handlingStrategy":"validation","validationCode":"if (tokens.LA(1) != Token.EOF) {\n  tokens.consume();\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never call consume() without first checking LA(1) != Token.EOF","For read-only scans use get(i)/size() instead of consume()","Write token loops as while (LA(1) != EOF), not while (true)"],"tags":["antlr","token-stream","eof","consume","off-by-one"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}