{"record":{"id":"f501750532a69674","repo":"antlr/antlr4","slug":"cannot-consume-eof-f50175","errorCode":null,"errorMessage":"cannot consume EOF","messagePattern":"cannot consume EOF","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java","lineNumber":139,"sourceCode":"\t\treturn \"\";\n\t}\n\n\n\t@Override\n\tpublic String getText(RuleContext ctx) {\n\t\treturn getText(ctx.getSourceInterval());\n\t}\n\n\n\t@Override\n\tpublic String getText(Token start, Token stop) {\n\t\treturn getText(Interval.of(start.getTokenIndex(), stop.getTokenIndex()));\n\t}\n\n\t@Override\n\tpublic void consume() {\n\t\tif (LA(1) == Token.EOF) {\n\t\t\tthrow new IllegalStateException(\"cannot consume EOF\");\n\t\t}\n\n\t\t// buf always has at least tokens[p==0] in this method due to ctor\n\t\tlastToken = tokens[p];   // track last token for LT(-1)\n\n\t\t// if we're at last token and no markers, opportunity to flush buffer\n\t\tif ( p == n-1 && numMarkers==0 ) {\n\t\t\tn = 0;\n\t\t\tp = -1; // p++ will leave this at 0\n\t\t\tlastTokenBufferStart = lastToken;\n\t\t}\n\n\t\tp++;\n\t\tcurrentTokenIndex++;\n\t\tsync(1);\n\t}\n\n\t/** Make sure we have 'need' elements from current position {@link #p p}. Last valid","sourceCodeStart":121,"sourceCodeEnd":157,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/UnbufferedTokenStream.java#L121-L157","documentation":"UnbufferedTokenStream.consume() first checks LA(1); when LA(1) is Token.EOF, the stream is positioned on the single end-of-input sentinel and there is no next token to consume. ANTLR therefore refuses to advance past EOF with an IllegalStateException. This is a stream-contract violation, not a normal syntax error.","triggerScenarios":"Calling consume() after the stream has reached EOF, including on an empty token source whose first token is EOF; custom loops that consume one token too many; error-recovery or listener code that consumes while recovering at EOF; or a custom TokenSource that returns EOF before the input expected by the parser.","commonSituations":"Hand-written token-stream drivers, custom parse error strategies, custom TokenSource implementations, and empty input files. Generated parsers normally stop at EOF, so a hit here usually means custom consumption logic or a token source that emits EOF at the wrong place.","solutions":["Check tokens.LA(1) != Token.EOF before every consume().","Fix custom while/for loops so EOF is the loop terminator and is never consumed.","If generated parsing reaches this point, debug the custom error strategy or TokenSource that advances past the sentinel.","Verify a custom TokenSource emits exactly one EOF token after all real tokens."],"exampleFix":"// before\nwhile (!done) {\n    process(tokens.LT(1));\n    tokens.consume(); // throws when LT(1) is EOF\n}\n\n// after\nwhile (tokens.LA(1) != Token.EOF) {\n    process(tokens.LT(1));\n    tokens.consume();\n}","handlingStrategy":"validation","validationCode":"if (tokens.LA(1) == Token.EOF) {\n    // no real token remains; do not call consume()\n}","typeGuard":null,"tryCatchPattern":"try {\n    tokens.consume();\n} catch (IllegalStateException e) {\n    if (tokens.LA(1) == Token.EOF) {\n        // treat as clean end of input\n    } else {\n        throw e;\n    }\n}","preventionTips":["Make Token.EOF an explicit loop terminator.","Never assume consume() is safe after LT(1) returns EOF.","Test custom token drivers with empty input and input whose last token is optional."],"tags":["antlr","java","token-stream","eof","unbuffered"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}