influxdata/influxdb · critical

ordered catalog batch should succeed when applied

Error message

ordered catalog batch should succeed when applied

What it means

Applying an ordered catalog batch does a double expect: the first asserts apply_catalog_batch never fails for a batch already known to be in sequence order, the second asserts the apply actually produced changes. Either panic means the batch was out of order, malformed, already applied (empty change set), or the underlying catalog/store rejected a batch the ordering check accepted.

Solutions

1. Verify batch sequence numbers are strictly sequential with no gaps or duplicates at the producer; fix the writer that assigned them
2. Inspect the persisted catalog batches around the failing sequence number for duplication or corruption and remove/repair them
3. Check for multiple processes/nodes writing to the same catalog concurrently; ensure only one writer owns sequence assignment
4. Reproduce with debug logging of apply_catalog_batch errors to identify the inner failure the expect swallowed
5. If a batch legitimately applies no changes, adjust the producer to skip empty batches

Example fix

// before
.inner.write()
.apply_catalog_batch(batch.batch(), batch.sequence_number(), Some(&self.store))
.expect("ordered catalog batch should succeed when applied")
.expect("ordered catalog batch should contain changes");
// after
let changes = inner
    .apply_catalog_batch(batch.batch(), batch.sequence_number(), Some(&self.store))
    .map_err(|e| anyhow!("batch seq {} failed to apply: {e}", batch.sequence_number()))?
    .ok_or_else(|| anyhow!("batch seq {} produced no changes", batch.sequence_number()))?;

// verify sequence continuity before applying
let expected = catalog.last_sequence_number() + 1;
if batch.sequence_number() != expected {
    return Err(anyhow!("out-of-order batch: got {}, expected {}",
        batch.sequence_number(), expected));
}

fn is_applicable(batch: &CatalogBatch, expected_seq: u64) -> bool {
    batch.sequence_number() == expected_seq
}

match inner.apply_catalog_batch(batch.batch(), seq, Some(&store)) {
    Ok(Some(changes)) => Ok(changes),
    Ok(None) => Err(anyhow!("batch {seq} produced no changes; duplicate?")),
    Err(e) => Err(anyhow!("batch {seq} failed to apply: {e}")),
}

When it happens

Trigger: Applying a CatalogBatch via the ordered apply path when (a) its sequence number is not the expected next sequence, (b) the batch fails validation inside apply_catalog_batch (bad column types, missing dependencies), or (c) the batch is a duplicate whose application yields no changes, tripping the second expect.

Common situations: Catalog replay after a crash with corrupted or duplicated persisted batches; multiple writers to the same catalog producing overlapping sequence numbers; upgrading across versions where persisted batch format changed.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• column id in series key should be valid
• Existing transaction for table should not exist
• no duplicate table by ID or name
• table should exist by id
• auto field family exists

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/b1f3ce8d717b97a9. Report an issue: GitHub.

Appendix: source

Thrown at influxdb3_catalog/src/catalog/versions/v2.rs:709

    /// This accepts a `_permit`, which is not used, and is just a way to ensure that the caller
    /// has a handle on the write permit at the time of invocation.
    pub(crate) fn apply_ordered_catalog_batch(
        &self,
        batch: &OrderedCatalogBatch,
        _permit: &CatalogWritePermit,
    ) -> CatalogBatch {
        let batch_sequence = batch.sequence_number().get();
        let current_sequence = self.sequence_number().get();
        assert_eq!(
            batch_sequence,
            current_sequence + 1,
            "catalog batch received out of order"
        );
        let catalog_batch = self
            .inner
            .write()
            .apply_catalog_batch(batch.batch(), batch.sequence_number(), Some(&self.store))
            .expect("ordered catalog batch should succeed when applied")
            .expect("ordered catalog batch should contain changes");
        self.update_last_check_time();
        catalog_batch.into_batch()
    }

    pub fn node(&self, node_id: &str) -> Option<Arc<NodeDefinition>> {
        self.inner.read().nodes.get_by_name(node_id)
    }

    pub fn node_by_id(&self, node_id: &NodeId) -> Option<Arc<NodeDefinition>> {
        self.inner.read().nodes.get_by_id(node_id)
    }

    pub fn list_nodes(&self) -> Vec<Arc<NodeDefinition>> {
        self.inner.read().nodes.resource_iter().cloned().collect()
    }

    pub fn minimum_supported_row_delete_predicate_version(&self) -> Option<usize> {

View on GitHub (pinned to 06200ef96b)