What causes "cannot consume EOF" in dbt-labs/dbt-core?

Check la(1)/current token before calling consume(); stop consuming once the token is EOF.

cannot consume EOF

Error message

cannot consume EOF

What it means

The ANTLR-style input stream's consume() advances the current index by one token via offset(). If offset() returns None the stream is already at EOF, and consuming past EOF is a lexer/parser invariant violation: no well-formed parse should ever ask to consume the EOF sentinel itself. This panic means the calling recognizer logic advanced one token too far.

Source

Thrown at crates/dbt-sql/dbt-sql-utils/src/input_streams.rs:62

            name: "<empty>".to_string(),
            data_raw,
            index: 0,
            // phantom: Default::default(),
        }
    }
}
impl<Data: Deref> IntStream for CaseInsensitiveInputStream<Data>
where
    Data::Target: InputData,
{
    #[inline]
    fn consume(&mut self) {
        if let Some(index) = self.data_raw.offset(self.index, 1) {
            self.index = index;
            // self.current = self.data_raw.deref().item(index).unwrap_or(TOKEN_EOF);
            // Ok(())
        } else {
            unreachable!("cannot consume EOF");
        }
    }

    #[inline]
    fn la(&mut self, mut offset: isize) -> i32 {
        assert!(offset != 0, "offset must not be 0");

        if offset == 1 {
            return match self.data_raw.item(self.index) {
                Some(v) => match v {
                    97..=122 => v - 32,
                    _ => v,
                },
                None => int_stream::EOF,
            };
        }
        if offset < 0 {
            offset += 1; // e.g., translate LA(-1) to use offset i=0; then data[p+0-1]

View on GitHub (pinned to 0267ce9170)

Solutions

Check la(1)/current token before calling consume(); stop consuming once the token is EOF.
Audit custom parsing loops for missing `if token == EOF { break }` guards.
Verify the input token stream is produced by the matching lexer so EOF is emitted exactly once at the end.
If reproducing on a specific SQL file, reduce the input to isolate which construct drives the parser past EOF.

Example fix

// before
while self.current() != TOKEN_EOF { self.consume(); }

// after
while let Some(tok) = self.la(1) { if tok == TOKEN_EOF { break; } self.consume(); }

Defensive patterns

Strategy: type-guard

Validate before calling

if self.la(1) == Some(TOKEN_EOF) { /* do not consume */ }

Type guard

fn can_consume(&self) -> bool {
    self.data_raw.offset(self.index, 1).is_some()
}

Try / catch

// consume only when a next token exists
if let Some(next) = self.data_raw.offset(self.index, 1) { self.index = next; } else { break; }

Prevention

Check la(1) != EOF before every consume() call.
In custom grammar/parse loops, break on the EOF sentinel.
Ensure the lexer emits exactly one trailing EOF token.

When it happens

Trigger: A parser/lexer loop calls consume() when la(1) is already EOF, or after recognizing a token stream that ends without the recognizer stopping at the EOF sentinel.

Common situations: Custom grammar rules that don't terminate on EOF; hand-written token consumption loops around dbt-sql-utils streams; changes to adaptivePredict or token production that shift the stream so EOF is reached earlier than the caller expects.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of dbt-labs/dbt-core@0267ce9170 (2026-09-07). Data as JSON: /api/errors/15d420c50ae67a6c. Report an issue: GitHub.