antlr/antlr4 · error · InvalidOperationException

cannot consume EOF

Error message

cannot consume EOF

What it means

UnbufferedCharStream is a forward-only character input. Consume() is illegal once LA(1) is EOF because there is no character left to consume. This guard protects the stream invariant that consumption only happens while real input remains.

Source

Thrown at runtime/CSharp/src/UnbufferedCharStream.cs:156

        {
            this.input = new StreamReader(input);
            Fill(1);
        }

        public UnbufferedCharStream(TextReader input, int bufferSize)
            : this(bufferSize)
        {
            // prime
            this.input = input;
            Fill(1);
        }

        // prime
        public virtual void Consume()
        {
            if (LA(1) == IntStreamConstants.EOF)
            {
                throw new InvalidOperationException("cannot consume EOF");
            }
            // buf always has at least data[p==0] in this method due to ctor
            lastChar = data[p];
            // track last char for LA(-1)
            if (p == n - 1 && numMarkers == 0)
            {
                n = 0;
                p = -1;
                // p++ will leave this at 0
                lastCharBufferStart = lastChar;
            }
            p++;
            currentCharIndex++;
            Sync(1);
        }

        /// <summary>
        /// Make sure we have 'need' elements from current position

View on GitHub (pinned to 7d5770395b)

Solutions

1. Check LA(1) != IntStreamConstants.EOF before every Consume().
2. Use a fresh stream for each parse.
3. Audit custom code that drives Consume() and ensure it stops at EOF.
4. If random access or repeated parsing is required, use AntlrInputStream with fully buffered input.

Example fix

// before
stream.Consume();

// after
if (stream.LA(1) != IntStreamConstants.EOF)
    stream.Consume();

if (stream.LA(1) != IntStreamConstants.EOF)
    stream.Consume();

try { stream.Consume(); }
catch (InvalidOperationException ex) when (ex.Message == "cannot consume EOF") { /* stop consuming */ }

When it happens

Trigger: User code manually calls charStream.Consume() after already reading to EOF; a custom lexer or test loop consumes without checking LA(1); or the same stream is exhausted by one consumer and then consumed again.

Common situations: Hand-written tokenizing loops; harnesses that call Consume() an input-length number of times plus one; reusing an UnbufferedCharStream after a previous parse; or custom ICharStream adapters with incorrect EOF behavior.

Related errors

• release() called with an invalid marker.
• cannot seek to negative index ${index}
• seek to index outside buffer: ${index} not in ${bufferStartI
• Unbuffered stream cannot know its size
• the interval extends past the end of the stream

AI-assisted analysis of antlr/antlr4@7d5770395b (2026-08-14). Data as JSON: /api/errors/4aadd89966958162. Report an issue: GitHub.