{"record":{"id":"fac36d83d27261f5","repo":"antlr/antlr4","slug":"cannot-consume-eof-fac36d","errorCode":null,"errorMessage":"cannot consume EOF","messagePattern":"cannot consume EOF","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"runtime/Java/src/org/antlr/v4/runtime/CodePointCharStream.java","lineNumber":96,"sourceCode":"\t\t\t\t\t\tname,\n\t\t\t\t\t\tcodePointBuffer.charArray(),\n\t\t\t\t\t\tcodePointBuffer.arrayOffset());\n\t\t\tcase INT:\n\t\t\t\treturn new CodePoint32BitCharStream(\n\t\t\t\t\t\tcodePointBuffer.position(),\n\t\t\t\t\t\tcodePointBuffer.remaining(),\n\t\t\t\t\t\tname,\n\t\t\t\t\t\tcodePointBuffer.intArray(),\n\t\t\t\t\t\tcodePointBuffer.arrayOffset());\n\t\t}\n\t\tthrow new UnsupportedOperationException(\"Not reached\");\n\t}\n\n\t@Override\n\tpublic final void consume() {\n\t\tif (size - position == 0) {\n\t\t\tassert LA(1) == IntStream.EOF;\n\t\t\tthrow new IllegalStateException(\"cannot consume EOF\");\n\t\t}\n\t\tposition = position + 1;\n\t}\n\n\t@Override\n\tpublic final int index() {\n\t\treturn position;\n\t}\n\n\t@Override\n\tpublic final int size() {\n\t\treturn size;\n\t}\n\n\t/** mark/release do nothing; we have entire buffer */\n\t@Override\n\tpublic final int mark() {\n\t\treturn -1;","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/antlr/antlr4/blob/7d5770395bb7b02eb56e7c62662cb1d7c08f42a3/runtime/Java/src/org/antlr/v4/runtime/CodePointCharStream.java#L78-L114","documentation":"CodePointCharStream.consume() throws IllegalStateException when position == size, i.e. the cursor sits on the EOF sentinel and there is nothing left to consume. This is ANTLR's IntStream contract: consume() past the end is a programming error, and LA(1) at that point returns IntStream.EOF. Lexers built on the stream never trigger this; hand-written stream manipulation does.","triggerScenarios":"Calling charStream.consume() in a custom loop after index() == size() - 1 has already been consumed once more; custom TokenSource implementations advancing a shared CharStream past the end; a Lexer subclass overriding nextToken() incorrectly.","commonSituations":"Manual scanning experiments over CharStream; porting code from another stream API that permits consuming at end; bugs in lexer wrappers that double-consume the final character.","solutions":["Check LA(1) != IntStream.EOF (or index() < size()) before each consume()","Prefer indexed reads (LA(i), index(), size()) over consume() for read-only scanning","Audit custom Lexer/TokenSource subclasses for consume() calls not guarded by an EOF check"],"exampleFix":"// before\nwhile (true) { process(cs.LA(1)); cs.consume(); } // throws at EOF\n\n// after\nwhile (cs.LA(1) != IntStream.EOF) { process(cs.LA(1)); cs.consume(); }","handlingStrategy":"validation","validationCode":"if (cs.index() < cs.size()) { // equivalently LA(1) != IntStream.EOF\n  cs.consume();\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check LA(1) != IntStream.EOF before every consume() on a CharStream","Use LA(k) lookahead reads instead of consume() when scanning without advancing","Keep a single owner for stream position; never let two components consume the same stream"],"tags":["antlr","char-stream","eof","consume"],"backgroundTag":null,"analyzedSha":"7d5770395bb7b02eb56e7c62662cb1d7c08f42a3","analyzedAt":"2026-08-14T14:47:56.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}