{"record":{"id":"fa2f9d58d4715aff","repo":"antlr/antlr4","slug":"cannot-consume-eof-fa2f9d","errorCode":null,"errorMessage":"cannot consume EOF","messagePattern":"cannot consume EOF","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java","lineNumber":123,"sourceCode":"\t\tthis(input, bufferSize, StandardCharsets.UTF_8);\n\t}\n\n\tpublic UnbufferedCharStream(InputStream input, int bufferSize, Charset charset) {\n\t\tthis(bufferSize);\n\t\tthis.input = new InputStreamReader(input, charset);\n\t\tfill(1); // prime\n\t}\n\n\tpublic UnbufferedCharStream(Reader input, int bufferSize) {\n\t\tthis(bufferSize);\n\t\tthis.input = input;\n\t\tfill(1); // prime\n\t}\n\n\t@Override\n\tpublic void consume() {\n\t\tif (LA(1) == IntStream.EOF) {\n\t\t\tthrow new IllegalStateException(\"cannot consume EOF\");\n\t\t}\n\n\t\t// buf always has at least data[p==0] in this method due to ctor\n\t\tlastChar = data[p];   // track last char for LA(-1)\n\n\t\tif (p == n-1 && numMarkers==0) {\n\t\t\tn = 0;\n\t\t\tp = -1; // p++ will leave this at 0\n\t\t\tlastCharBufferStart = lastChar;\n\t\t}\n\n\t\tp++;\n\t\tcurrentCharIndex++;\n\t\tsync(1);\n\t}\n\n\t/**\n\t * Make sure we have 'need' elements from current position {@link #p p}.","sourceCodeStart":105,"sourceCodeEnd":141,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/UnbufferedCharStream.java#L105-L141","documentation":"UnbufferedCharStream.consume() throws IllegalStateException when LA(1) is EOF, because consuming past the end of the character input is a protocol violation for ANTLR streams. Unlike buffered streams, the unbuffered stream keeps only a small window, so it checks the current lookahead symbol before advancing.","triggerScenarios":"Calling stream.consume() when LA(1) already returned IntStream.EOF (-1); a hand-written scanner loop that consumes without checking for EOF; a Lexer subclass whose nextToken() logic consumes after EOF was reached.","commonSituations":"Custom lexer code or manual CharStream iteration on large files where UnbufferedCharStream was chosen to save memory; mismatch between the generated lexer's EOF handling and added custom consume() calls.","solutions":["Check LA(1) != IntStream.EOF before every consume()","In custom lexer overrides, follow the generated nextToken() pattern which stops at EOF","If you need unrestricted lookbehind/re-reads, use ANTLRInputStream (buffered) instead of UnbufferedCharStream"],"exampleFix":"// before\nwhile (true) { stream.consume(); } // throws at EOF\n\n// after\nwhile (stream.LA(1) != IntStream.EOF) {\n    processChar(stream.LA(1));\n    stream.consume();\n}","handlingStrategy":"validation","validationCode":"if (stream.LA(1) != IntStream.EOF) {\n    stream.consume();\n}","typeGuard":null,"tryCatchPattern":"try { stream.consume(); } catch (IllegalStateException e) { /* at EOF: stop scanning */ }","preventionTips":["Every consume() is preceded by an LA(1) EOF check","Model custom lexers on the generated nextToken() loop","Choose buffered streams when you need re-read semantics"],"tags":["antlr","char-stream","eof","consume"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}