What causes "Must reach the buffer end" in databendlabs/databend?

Inspect the remaining bytes (Cursor::split(self).1) to see what trailing data was left.

Must reach the buffer end

Error message

Must reach the buffer end

What it means

CursorReadBytesExt::must_eof asserts the cursor has consumed its entire underlying buffer. If unread bytes remain after a parse, it raises an InvalidData io error, enforcing that the input contained nothing beyond the expected structure.

Solutions

Inspect the remaining bytes (Cursor::split(self).1) to see what trailing data was left.
Explicitly consume/ignore allowed trailing content (e.g. whitespace) before calling must_eof.
Fix the parser to consume all expected tokens, or relax to eof() if trailing data is acceptable.

Example fix

// before
cursor.must_eof()?;

// after
cursor.keep_read(&mut Vec::new(), |b| b.is_ascii_whitespace());
cursor.must_eof()?;

Defensive patterns

Strategy: validation

Validate before calling

if !cursor.eof() {
    return Err(anyhow!("trailing input after parsed value"));
}

Try / catch

if let Err(e) = cursor.must_eof() {
    return Err(anyhow!("trailing data: {}", e));
}

Prevention

Trim or explicitly consume allowed trailing whitespace before must_eof.
Use the boolean eof() when trailing content is acceptable.
Log remaining bytes on failure to identify unconsumed input.

When it happens

Trigger: Calling must_eof() when the cursor still holds unconsumed bytes — e.g. trailing characters/garbage after the parsed value or an incomplete parse that stopped early.

Common situations: Strict parsers validating user-supplied strings or binary payloads where trailing whitespace, extra tokens, or wrong-format suffixes remain unconsumed.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/70af3cc4fcd5830e. Report an issue: GitHub.

Appendix: source

Thrown at src/common/io/src/cursor_ext/cursor_read_bytes_ext.rs:90

        let buf = Cursor::split(self).1;
        if buf.is_empty() {
            None
        } else {
            Some(buf[0] as char)
        }
    }

    fn peek_byte(&self) -> Option<u8> {
        let buf = Cursor::split(self).1;
        if buf.is_empty() { None } else { Some(buf[0]) }
    }

    fn eof(&mut self) -> bool {
        Cursor::split(self).1.is_empty()
    }
    fn must_eof(&mut self) -> Result<()> {
        if !Cursor::split(self).1.is_empty() {
            return Err(std::io::Error::new(
                ErrorKind::InvalidData,
                "Must reach the buffer end",
            ));
        }
        Ok(())
    }

    fn ignore(&mut self, f: impl Fn(u8) -> bool) -> bool {
        let available = Cursor::split(self).1;
        if available.is_empty() {
            false
        } else if f(available[0]) {
            self.consume(1);
            true
        } else {
            false
        }
    }

View on GitHub (pinned to 288d84d76e)