antlr/antlr4 · error · InvalidOperationException

cannot consume EOF

Error message

cannot consume EOF

What it means

BufferedTokenStream.Consume() advances the stream pointer by one token, but the token stream is logically terminated by a single EOF token; consuming EOF would move the index past the end of the stream, which is meaningless. The check fires when LA(1) already returns IntStreamConstants.EOF and the skip-EofCheck optimization does not apply (i.e. the caller has not pre-fetched beyond the current position).

Source

Thrown at runtime/CSharp/src/BufferedTokenStream.cs:189

                {
                    // the last token in tokens is EOF. skip check if p indexes any
                    // fetched token except the last.
                    skipEofCheck = p < tokens.Count - 1;
                }
                else
                {
                    // no EOF token in tokens. skip check if p indexes a fetched token.
                    skipEofCheck = p < tokens.Count;
                }
            }
            else
            {
                // not yet initialized
                skipEofCheck = false;
            }
            if (!skipEofCheck && LA(1) == IntStreamConstants.EOF)
            {
                throw new InvalidOperationException("cannot consume EOF");
            }
            if (Sync(p + 1))
            {
                p = AdjustSeekIndex(p + 1);
            }
        }

        /// <summary>
        /// Make sure index
        /// <paramref name="i"/>
        /// in tokens has a token.
        /// </summary>
        /// <returns>
        ///
        /// <see langword="true"/>
        /// if a token is located at index
        /// <paramref name="i"/>
        /// , otherwise

View on GitHub (pinned to 7d5770395b)

Solutions

1. Guard every manual Consume() with a check that LA(1) != IntStreamConstants.EOF
2. If you have a custom ITokenSource, ensure it emits exactly one EOF token and never emits tokens after EOF
3. Do not consume tokens from listener/visitor callbacks while the generated parser is driving the same stream

Example fix

// before
while (true) { Process(stream.LT(1)); stream.Consume(); } // throws at EOF

// after
while (stream.LA(1) != IntStreamConstants.EOF)
{
    Process(stream.LT(1));
    stream.Consume();
}

if (stream.LA(1) == IntStreamConstants.EOF)
    return; // nothing left to consume
stream.Consume();

static bool CanConsume(ITokenStream s) => s.LA(1) != IntStreamConstants.EOF;

When it happens

Trigger: Calling Consume() while the current lookahead token is EOF: a parser or manual loop that consumes one token too many; custom ITokenSource implementations that emit EOF more than once or at the wrong time (the fetchedEOF flag then disagrees with the source); calling Consume() after the parser already reported EOF as part of an error-recovery path.

Common situations: Hand-written token loops like while (stream.LA(1) != Eof) { ...; stream.Consume(); } that also consume inside the body; custom lexers that emit multiple EOF tokens; parsers or listeners that consume tokens themselves while ANTLR's generated code also consumes.

Related errors

• cannot consume EOF
• seek to index outside buffer: ${index} not in ${bufferStartI
• cannot consume EOF
• cannot consume EOF
• This ATN simulator does not support clearing the DFA.

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/9d353e0408be516a. Report an issue: GitHub.