risingwavelabs/risingwave · critical · BackupError

inconsistent hummock version: expected {}, actual {}

Error message

inconsistent hummock version: expected {}, actual {}

What it means

While building a meta snapshot, the builder replays hummock version deltas on top of a base version; after replay it checks that the highest applied delta log id equals the resulting redo state's version id. A mismatch means the persisted version chain is inconsistent (deltas missing, duplicated, or from a diverged history), so the backup is aborted.

Source

Thrown at src/meta/src/backup_restore/meta_snapshot_builder.rs:106

            .all(&txn)
            .await
            .map_err(map_db_err)?
            .into_iter()
            .map_into::<PbHummockVersionDelta>()
            .map(HummockVersionDelta::from_persisted_protobuf_owned);
        let hummock_version = {
            let mut redo_state = hummock_version;
            let mut max_log_id = None;
            for version_delta in version_deltas {
                if version_delta.prev_id == redo_state.id {
                    redo_state.apply_version_delta(&version_delta);
                }
                max_log_id = Some(version_delta.id);
            }
            if let Some(max_log_id) = max_log_id
                && max_log_id != redo_state.id
            {
                return Err(BackupError::Other(anyhow::anyhow!(format!(
                    "inconsistent hummock version: expected {}, actual {}",
                    max_log_id, redo_state.id
                ))));
            }
            redo_state
        };
        let mut metadata = MetadataV2 {
            hummock_version,
            ..Default::default()
        };
        set_metadata(&mut metadata, &txn).await?;

        txn.commit().await.map_err(map_db_err)?;
        self.snapshot.metadata = metadata;
        Ok(())
    }

    pub fn finish(self) -> BackupResult<MetaSnapshotV2> {

View on GitHub (pinned to 6469eb736d)

Solutions

1. Retry the backup after ensuring no concurrent hummock version writes are in flight
2. Inspect the meta store's hummock version delta logs for corruption or gaps
3. Restore from a known-good backup and re-run; escalate as a data-consistency issue if reproducible

// Pre-check hummock delta log chain integrity before building snapshot
assert_eq!(max_delta_log_id, redo_state.id, "version chain inconsistent");

match builder.build().await {
    Err(BackupError::Other(e)) if e.to_string().contains("inconsistent hummock version") => {
        // abort backup, inspect meta store, retry after quiescing writes
    }
    other => other?,
}

When it happens

Trigger: Replaying version deltas during MetaSnapshotBuilder::build where max_replayed delta log id != redo_state.id — e.g. corrupted/truncated delta logs or concurrent hummock writes racing with snapshot creation.

Common situations: Corrupted meta store or backup source; concurrent writes to hummock version deltas while a snapshot is being taken; restoring from a partially written backup.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• job {} in database {} has tables with different table ids. {
• concurrent backup job is not supported: existent job {}
• too many existent meta snapshots, expect at most {}
• snapshot id {} not found
• worker_id {} for actor {} does not exist

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/c3ac64a38681fd92. Report an issue: GitHub.