antlr/antlr4 · error · InvalidOperationException

cannot consume EOF

Error message

cannot consume EOF

What it means

UnbufferedTokenStream.Consume() requires a real token at the current position. If LA(1) is EOF, the token stream has reached end of input and consuming again is illegal. This mirrors the EOF guard on UnbufferedCharStream.Consume().

Source

Thrown at runtime/CSharp/src/UnbufferedTokenStream.cs:190

        {
            return GetText(ctx.SourceInterval);
        }

        [return: NotNull]
        public virtual string GetText(IToken start, IToken stop)
        {
            if (start != null && stop != null)
            {
                return GetText(Interval.Of(start.TokenIndex, stop.TokenIndex));
            }
            throw new NotSupportedException("The specified start and stop symbols are not supported.");
        }

        public virtual void Consume()
        {
            if (LA(1) == TokenConstants.EOF)
            {
                throw new InvalidOperationException("cannot consume EOF");
            }
            // buf always has at least tokens[p==0] in this method due to ctor
            lastToken = tokens[p];
            // track last token for LT(-1)
            // if we're at last token and no markers, opportunity to flush buffer
            if (p == n - 1 && numMarkers == 0)
            {
                n = 0;
                p = -1;
                // p++ will leave this at 0
                lastTokenBufferStart = lastToken;
            }
            p++;
            currentTokenIndex++;
            Sync(1);
        }

        /// <summary>

View on GitHub (pinned to 7d5770395b)

Solutions

1. Check LT(1).Type != TokenConstants.EOF before every Consume().
2. Break token loops immediately when EOF is seen.
3. Create a new lexer/token stream for each parse rather than reusing an exhausted one.

Example fix

// before
tokens.Consume();

// after
if (tokens.LT(1).Type != TokenConstants.EOF)
    tokens.Consume();

if (tokens.LT(1).Type != TokenConstants.EOF)
    tokens.Consume();

try { tokens.Consume(); }
catch (InvalidOperationException ex) when (ex.Message == "cannot consume EOF") { /* end token loop */ }

When it happens

Trigger: A token loop calls Consume() once too often; Consume() is called after LT(1) already returned EOF; or the same UnbufferedTokenStream is reused after exhaustion.

Common situations: Manual iteration over tokens with inclusive bounds; parse loops that do not break on EOF; harnesses that consume Size+1 tokens; custom error recovery that continues after EOF.

Related errors

• cannot consume EOF
• cannot consume EOF
• get(${i}) outside buffer: ${bufferStartIndex}..${bufferStart
• seek to index outside buffer: ${index} not in ${bufferStartI
• Unbuffered stream cannot know its size

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/42afe71468596524. Report an issue: GitHub.