{"record":{"id":"c3ac64a38681fd92","repo":"risingwavelabs/risingwave","slug":"inconsistent-hummock-version-expected-actual","errorCode":null,"errorMessage":"inconsistent hummock version: expected {}, actual {}","messagePattern":"inconsistent hummock version: expected (.+?), actual (.+?)","errorType":"exception","errorClass":"BackupError","httpStatus":null,"severity":"critical","filePath":"src/meta/src/backup_restore/meta_snapshot_builder.rs","lineNumber":106,"sourceCode":"            .all(&txn)\n            .await\n            .map_err(map_db_err)?\n            .into_iter()\n            .map_into::<PbHummockVersionDelta>()\n            .map(HummockVersionDelta::from_persisted_protobuf_owned);\n        let hummock_version = {\n            let mut redo_state = hummock_version;\n            let mut max_log_id = None;\n            for version_delta in version_deltas {\n                if version_delta.prev_id == redo_state.id {\n                    redo_state.apply_version_delta(&version_delta);\n                }\n                max_log_id = Some(version_delta.id);\n            }\n            if let Some(max_log_id) = max_log_id\n                && max_log_id != redo_state.id\n            {\n                return Err(BackupError::Other(anyhow::anyhow!(format!(\n                    \"inconsistent hummock version: expected {}, actual {}\",\n                    max_log_id, redo_state.id\n                ))));\n            }\n            redo_state\n        };\n        let mut metadata = MetadataV2 {\n            hummock_version,\n            ..Default::default()\n        };\n        set_metadata(&mut metadata, &txn).await?;\n\n        txn.commit().await.map_err(map_db_err)?;\n        self.snapshot.metadata = metadata;\n        Ok(())\n    }\n\n    pub fn finish(self) -> BackupResult<MetaSnapshotV2> {","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/backup_restore/meta_snapshot_builder.rs#L88-L124","documentation":"While building a meta snapshot, the builder replays hummock version deltas on top of a base version; after replay it checks that the highest applied delta log id equals the resulting redo state's version id. A mismatch means the persisted version chain is inconsistent (deltas missing, duplicated, or from a diverged history), so the backup is aborted.","triggerScenarios":"Replaying version deltas during MetaSnapshotBuilder::build where max_replayed delta log id != redo_state.id — e.g. corrupted/truncated delta logs or concurrent hummock writes racing with snapshot creation.","commonSituations":"Corrupted meta store or backup source; concurrent writes to hummock version deltas while a snapshot is being taken; restoring from a partially written backup.","solutions":["Retry the backup after ensuring no concurrent hummock version writes are in flight","Inspect the meta store's hummock version delta logs for corruption or gaps","Restore from a known-good backup and re-run; escalate as a data-consistency issue if reproducible"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-check hummock delta log chain integrity before building snapshot\nassert_eq!(max_delta_log_id, redo_state.id, \"version chain inconsistent\");","typeGuard":null,"tryCatchPattern":"match builder.build().await {\n    Err(BackupError::Other(e)) if e.to_string().contains(\"inconsistent hummock version\") => {\n        // abort backup, inspect meta store, retry after quiescing writes\n    }\n    other => other?,\n}","preventionTips":["Avoid triggering snapshots while hummock version deltas are being written concurrently","Monitor meta store health and delta log integrity","Keep backups of the meta store itself to recover from corruption"],"tags":["rust","meta","backup","hummock","consistency"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}