{"record":{"id":"3b1dd887f11c8769","repo":"quickwit-oss/tantivy","slug":"expunrolledlinkedlist-block-count-overflow-excee","errorCode":null,"errorMessage":"ExpUnrolledLinkedList block count overflow - exceeded 4 billion blocks","messagePattern":"ExpUnrolledLinkedList block count overflow - exceeded 4 billion blocks","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"stacker/src/expull.rs","lineNumber":133,"sourceCode":"// Block size caps at 32KB (2^15) regardless of how high block_num goes\n#[inline]\nfn get_block_size(block_num: u32) -> u16 {\n    // Cap at 15 to prevent block sizes > 32KB\n    // block_num can now be much larger than 15, but block size maxes out\n    let exp: u32 = block_num.min(15u32);\n    (1u32 << exp) as u16\n}\n\nimpl ExpUnrolledLinkedList {\n    #[inline(always)]\n    pub fn increment_num_blocks(&mut self) {\n        // Add overflow check as a safety measure\n        // With u32, we can handle up to ~4 billion blocks before overflow\n        // At 32KB per block (max size), that's 128 TB of data\n        self.block_num = self\n            .block_num\n            .checked_add(1)\n            .expect(\"ExpUnrolledLinkedList block count overflow - exceeded 4 billion blocks\");\n    }\n\n    #[inline]\n    pub fn writer<'a>(&'a mut self, arena: &'a mut MemoryArena) -> ExpUnrolledLinkedListWriter<'a> {\n        ExpUnrolledLinkedListWriter { eull: self, arena }\n    }\n\n    pub fn read_to_end(&self, arena: &MemoryArena, output: &mut Vec<u8>) {\n        let mut addr = self.head;\n        if addr.is_null() {\n            return;\n        }\n\n        // Calculate last block length with bounds checking to prevent underflow\n        let block_size = get_block_size(self.block_num) as usize;\n        let last_block_len = block_size.saturating_sub(self.remaining_cap as usize);\n\n        // Safety check: if remaining_cap > block_size, the metadata is corrupted","sourceCodeStart":115,"sourceCodeEnd":151,"githubUrl":"https://github.com/quickwit-oss/tantivy/blob/b5d8deb80c26924e6b007a5b1a7630f35ca64de4/stacker/src/expull.rs#L115-L151","documentation":"`ExpUnrolledLinkedList::increment_num_blocks` stores the block count in a `u32` and uses `checked_add(1)` before bumping it. If the count would exceed `u32::MAX` (~4.29 billion blocks), the `expect` panics. Given the capped block size (max ~32KB per block), reaching this limit implies an astronomically large structure (~128 TB), so hitting it almost always indicates runaway unbounded writes into a shared arena-backed EULL rather than legitimate volume.","triggerScenarios":"Calling `increment_num_blocks` after 4,294,967,295 blocks have been allocated — reachable only through extreme sustained writes into one `ExpUnrolledLinkedList` (e.g. one giant posting list / term dictionary building), or via the stress test `test_eull_limit` exercising the limit directly.","commonSituations":"Very large index builds where a single term's posting list grows unbounded for weeks/months of accumulation, or a bug leaking growth into one EULL (e.g. indexing every document under one term), or adversarial data causing pathological term growth.","solutions":["Shard the data across multiple indexes/segments so no single unrolled linked list approaches billions of blocks.","Reduce posting-list size per term: remove over-general terms (e.g. a catch-all tag) or cap term cardinality at the application level.","Monitor memory/disk usage of index builds and abort builds that approach the limit instead of letting the panic surface mid-write.","If truly exceeding the limit, you need a different storage structure/version of the library with a wider block counter; upgrade tantivy or file an issue."],"exampleFix":"// before: everything funnels into one term's posting list\ndoc.add_text_value(catch_all_field, \"ALL\"); // single giant EULL\n\n// after: partition writes across shards/fields so each EULL stays small\nlet shard = doc_id % num_shards;\nwriters[shard].add_document(doc);","handlingStrategy":"validation","validationCode":"const MAX_SAFE_BLOCKS: u32 = 1_000_000_000; // far below u32::MAX\nfn eull_within_budget(block_num: u32) -> Result<(), String> {\n    if block_num >= MAX_SAFE_BLOCKS {\n        return Err(format!(\"EULL block count {} near overflow; shard the data\", block_num));\n    }\n    Ok(())\n}","typeGuard":"fn block_count_safe(block_num: u32) -> bool {\n    block_num < u32::MAX - 1\n}","tryCatchPattern":"let grew = std::panic::catch_unwind(|| eull.increment_num_blocks())\n    .map_err(|_| anyhow::anyhow!(\"EULL block overflow — abort build and shard the index\"))?;","preventionTips":["Avoid funneling unbounded writes into a single term's posting list (no catch-all terms)","Shard large datasets across multiple indexes/segments so no single EULL grows unbounded","Monitor arena/memory growth during long-running index builds and alert well before billions of blocks"],"tags":["panic","overflow","memory","indexing"],"backgroundTag":"integer-overflow","analyzedSha":"b5d8deb80c26924e6b007a5b1a7630f35ca64de4","analyzedAt":"2026-09-05T13:20:51.521Z","contentChangedAt":"2026-09-05T13:20:51.521Z","schemaVersion":2},"datasetVersion":"2026-09-12T17:17:11.597Z"}