quickwit-oss/tantivy · error

ExpUnrolledLinkedList block count overflow - exceeded 4 bill

Error message

ExpUnrolledLinkedList block count overflow - exceeded 4 billion blocks

What it means

`ExpUnrolledLinkedList::increment_num_blocks` stores the block count in a `u32` and uses `checked_add(1)` before bumping it. If the count would exceed `u32::MAX` (~4.29 billion blocks), the `expect` panics. Given the capped block size (max ~32KB per block), reaching this limit implies an astronomically large structure (~128 TB), so hitting it almost always indicates runaway unbounded writes into a shared arena-backed EULL rather than legitimate volume.

Source

Thrown at stacker/src/expull.rs:133

// Block size caps at 32KB (2^15) regardless of how high block_num goes
#[inline]
fn get_block_size(block_num: u32) -> u16 {
    // Cap at 15 to prevent block sizes > 32KB
    // block_num can now be much larger than 15, but block size maxes out
    let exp: u32 = block_num.min(15u32);
    (1u32 << exp) as u16
}

impl ExpUnrolledLinkedList {
    #[inline(always)]
    pub fn increment_num_blocks(&mut self) {
        // Add overflow check as a safety measure
        // With u32, we can handle up to ~4 billion blocks before overflow
        // At 32KB per block (max size), that's 128 TB of data
        self.block_num = self
            .block_num
            .checked_add(1)
            .expect("ExpUnrolledLinkedList block count overflow - exceeded 4 billion blocks");
    }

    #[inline]
    pub fn writer<'a>(&'a mut self, arena: &'a mut MemoryArena) -> ExpUnrolledLinkedListWriter<'a> {
        ExpUnrolledLinkedListWriter { eull: self, arena }
    }

    pub fn read_to_end(&self, arena: &MemoryArena, output: &mut Vec<u8>) {
        let mut addr = self.head;
        if addr.is_null() {
            return;
        }

        // Calculate last block length with bounds checking to prevent underflow
        let block_size = get_block_size(self.block_num) as usize;
        let last_block_len = block_size.saturating_sub(self.remaining_cap as usize);

        // Safety check: if remaining_cap > block_size, the metadata is corrupted

View on GitHub (pinned to b5d8deb80c)

Solutions

1. Shard the data across multiple indexes/segments so no single unrolled linked list approaches billions of blocks.
2. Reduce posting-list size per term: remove over-general terms (e.g. a catch-all tag) or cap term cardinality at the application level.
3. Monitor memory/disk usage of index builds and abort builds that approach the limit instead of letting the panic surface mid-write.
4. If truly exceeding the limit, you need a different storage structure/version of the library with a wider block counter; upgrade tantivy or file an issue.

Example fix

// before: everything funnels into one term's posting list
doc.add_text_value(catch_all_field, "ALL"); // single giant EULL

// after: partition writes across shards/fields so each EULL stays small
let shard = doc_id % num_shards;
writers[shard].add_document(doc);

const MAX_SAFE_BLOCKS: u32 = 1_000_000_000; // far below u32::MAX
fn eull_within_budget(block_num: u32) -> Result<(), String> {
    if block_num >= MAX_SAFE_BLOCKS {
        return Err(format!("EULL block count {} near overflow; shard the data", block_num));
    }
    Ok(())
}

fn block_count_safe(block_num: u32) -> bool {
    block_num < u32::MAX - 1
}

let grew = std::panic::catch_unwind(|| eull.increment_num_blocks())
    .map_err(|_| anyhow::anyhow!("EULL block overflow — abort build and shard the index"))?;

When it happens

Trigger: Calling `increment_num_blocks` after 4,294,967,295 blocks have been allocated — reachable only through extreme sustained writes into one `ExpUnrolledLinkedList` (e.g. one giant posting list / term dictionary building), or via the stress test `test_eull_limit` exercising the limit directly.

Common situations: Very large index builds where a single term's posting list grows unbounded for weeks/months of accumulation, or a bug leaking growth into one EULL (e.g. indexing every document under one term), or adversarial data causing pathological term growth.

Related errors

• non-zero packed bits fit in u64
• Failed to acquire write lock on delete queue writer
• Empty DocOpstamp is forbidden
• This lock should never be poisoned
• Failed to acquire read lock on SegmentManager.

AI-assisted analysis of quickwit-oss/tantivy@b5d8deb80c (2026-09-05). Data as JSON: /api/errors/3b1dd887f11c8769. Report an issue: GitHub.