quickwit-oss/tantivy · error · io::Error

failed to read block_len

Error message

failed to read block_len

What it means

Thrown by the SSTable block reader's read_block() when the encoded block_len field is 1..=3 bytes. A legal block length prefix is either 0 (special/last marker) or 4 bytes; a 1-3 byte length is structurally impossible, so the reader reports UnexpectedEof — the stream is misaligned or truncated.

Source

Thrown at sstable/src/block_reader.rs:88

        self.buffer.clear();

        loop {
            let block_len = match self.reader.len() {
                0 => {
                    // we are out of data for this block. Check if we have another block after
                    match self.next_readers.next() {
                        Some((new_reader, first_ordinal)) => {
                            self.reader = new_reader;
                            self.pending_first_ordinal = Some(first_ordinal);
                            continue;
                        }
                        _ => {
                            return Ok(false);
                        }
                    }
                }
                1..=3 => {
                    return Err(io::Error::new(
                        io::ErrorKind::UnexpectedEof,
                        "failed to read block_len",
                    ));
                }
                _ => self.reader.read_u32() as usize,
            };
            if block_len <= 1 {
                return Ok(false);
            }
            let compress = self.reader.read_u8();
            let block_len = block_len - 1;

            if self.reader.len() < block_len {
                return Err(io::Error::new(
                    io::ErrorKind::UnexpectedEof,
                    "failed to read block content",
                ));
            }

View on GitHub (pinned to b5d8deb80c)

Solutions

  1. Verify the SSTable file is complete and uncorrupted (compare size/checksum with the source)
  2. Rebuild the index or restore the SSTable from backup
  3. Ensure the reader starts at a valid block boundary offset (e.g. from the index/dictionary) rather than an arbitrary position
  4. Check writer/reader format version compatibility

Example fix

// before
match self.reader.read_bytes(1)? {
    0 => /* ... */,
    1..=3 => return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "failed to read block_len")),
    _ => self.reader.read_u32() as usize,
}
// after
match self.reader.read_bytes(1)? {
    0 => /* ... */,
    1..=3 => return Err(io::Error::new(io::ErrorKind::UnexpectedEof, format!("failed to read block_len: invalid length prefix at offset {}", self.reader.pos()))),
    _ => self.reader.read_u32() as usize,
}
Defensive patterns

Strategy: validation

Validate before calling

fn sstable_offset_is_block_aligned(reader: &BlockReader, offset: usize) -> bool {
    // a valid block starts with a length prefix of 0 or 4
    if offset >= reader.len() { return false; }
    match reader.peek_u8_at(offset) {
        Some(0) | Some(4) => true,
        _ => false,
    }
}

Type guard

fn has_valid_block_len_prefix(byte: u8) -> bool {
    byte == 0 || byte == 4
}

Try / catch

match block_reader.read_block() {
    Err(e) if e.kind() == io::ErrorKind::UnexpectedEof
        && e.to_string() == "failed to read block_len" => {
        eprintln!("SSTable truncated or offset misaligned; restoring from backup");
    }
    Err(e) => return Err(e.into()),
    Ok(done) => if !done { /* process block */ },
}

Prevention

When it happens

Trigger: Reading an SSTable stream where the next block_len byte count lands on 1..=3; this occurs with truncated files or when the read offset drifts into the middle of a length prefix due to prior misparses.

Common situations: Corrupted or truncated SSTable files; reading an SSTable written by an incompatible format version; incorrect seek/offset handling by the caller leading to mid-token reads.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/bd39cb2ff72e9448. Report an issue: GitHub.