quickwit-oss/tantivy · error · io::Error

error when reading block in doc store

Error message

error when reading block in doc store

What it means

Thrown in iter_raw() when reading a doc store block fails; the underlying io::ErrorKind from the block read is preserved but the message is replaced with the generic "error when reading block in doc store". It is reached through iter() while streaming raw blocks, so any block-read failure (missing block, IO error, corruption surfaced as an error kind) ends up here.

Source

Thrown at src/store/reader.rs:345

                    .unwrap_or(true);
                let res = if alive {
                    Some((curr_block.clone(), doc_pos))
                } else {
                    None
                };
                doc_pos += 1;
                res
            })
            .map(move |(block, doc_pos)| {
                let block = block
                    .ok_or_else(|| {
                        DataCorruption::comment_only(
                            "the current checkpoint in the doc store iterator is none, this \
                             should never happen",
                        )
                    })?
                    .map_err(|error_kind| {
                        std::io::Error::new(error_kind, "error when reading block in doc store")
                    })?;

                let range = block_read_index(&block, doc_pos)?;
                Ok(block.slice(range))
            })
    }

    /// Summarize total space usage of this store reader.
    pub fn space_usage(&self) -> StoreSpaceUsage {
        self.space_usage.clone()
    }
}

fn block_read_index(block: &[u8], doc_pos: u32) -> crate::Result<Range<usize>> {
    let doc_pos = doc_pos as usize;
    let size_of_u32 = std::mem::size_of::<u32>();

    let index_len_pos = block.len() - size_of_u32;

View on GitHub (pinned to b5d8deb80c)

Solutions

  1. Log the io::ErrorKind (preserved in error.kind()) to identify the root cause before the message is flattened
  2. Restore or rebuild the corrupted segment; verify checkpoints match existing blocks
  3. Check file permissions and disk health on the index directory
  4. Re-open the index fresh to rule out stale in-memory checkpoint state

Example fix

// before
let block = checkpoint.block_addr.map_err(|error_kind| std::io::Error::new(error_kind, "error when reading block in doc store"))?;
// after
let block = checkpoint.block_addr.map_err(|error_kind| std::io::Error::new(error_kind.clone(), format!("error when reading block in doc store (kind={error_kind:?})")))?;
Defensive patterns

Strategy: try-catch

Validate before calling

fn checkpoint_is_resolvable(checkpoint: &Checkpoint, store: &DocStoreReader) -> bool {
    match &checkpoint.block_addr {
        Some(addr) => addr.block_id < store.num_blocks(),
        None => false, // none-checkpoint would trip the internal comment-only corruption
    }
}

Type guard

fn has_valid_checkpoint<T>(checkpoint: &Option<T>) -> bool {
    checkpoint.is_some()
}

Try / catch

match doc_store_iter.next() {
    Err(e) => {
        // the original io::ErrorKind is preserved
        eprintln!("doc store block read failed with kind={:?}: {e}", e.kind());
        if e.kind() == std::io::ErrorKind::InvalidData {
            // corruption: rebuild or restore segment
        } else {
            // transient IO: safe to retry with backoff
        }
    }
    Ok(item) => process(item),
}

Prevention

When it happens

Trigger: Iterating doc store blocks via iter()/iter_raw() when the underlying block lookup returns an Err(io::ErrorKind), e.g. reading past the last block, a checkpoint pointing at a nonexistent block, or IO failures on the underlying file.

Common situations: Iterating a doc store whose checkpoint/block index is inconsistent (e.g. after a failed commit or partially deleted segment); disk/permission errors while streaming; consuming an iterator beyond the valid range of a corrupted segment.

Related errors

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/2efdacc126a7959b. Report an issue: GitHub.