{"record":{"id":"871af2e4875afe19","repo":"risingwavelabs/risingwave","slug":"invalid-time-travel-delta-chain-delta-has-prev","errorCode":null,"errorMessage":"invalid time travel delta chain: delta {} has prev version {}, but replay has reached {}","messagePattern":"invalid time travel delta chain: delta (.+?) has prev version (.+?), but replay has reached (.+?)","errorType":"exception","errorClass":"TimeTravel","httpStatus":null,"severity":"error","filePath":"src/meta/src/hummock/manager/time_travel.rs","lineNumber":856,"sourceCode":"}\n\n/// The `HummockVersion` is actually `InHummockVersion`. It requires `refill_version`.\nfn replay_archive(\n    version: PbHummockVersion,\n    deltas: impl Iterator<Item = PbHummockVersionDelta>,\n) -> Result<HummockVersion> {\n    // The pb version ann pb version delta are actually written by InHummockVersion and InHummockVersionDelta, respectively.\n    // Using HummockVersion make it easier for `refill_version` later.\n    let mut last_version = HummockVersion::from_persisted_protobuf_owned(version);\n    for d in deltas {\n        let d = HummockVersionDelta::from_persisted_protobuf_owned(d);\n        debug_assert!(\n            !should_mark_next_time_travel_version_snapshot(&d),\n            \"unexpected time travel delta {:?}\",\n            d\n        );\n        if d.prev_id < last_version.id {\n            return Err(Error::TimeTravel(anyhow!(format!(\n                \"invalid time travel delta chain: delta {} has prev version {}, but replay has reached {}\",\n                d.id, d.prev_id, last_version.id\n            ))));\n        }\n        // Compaction deltas are not included in the time travel archive, so there may be gaps\n        // between the last replayed version and this delta's previous version.\n        last_version.id = d.prev_id;\n        last_version.apply_version_delta(&d);\n    }\n    Ok(last_version)\n}\n\npub fn require_sql_meta_store_err() -> Error {\n    Error::TimeTravel(anyhow!(\"require SQL meta store\"))\n}\n\n/// Time travel delta replay only expect `NewL0SubLevel`. In all other cases, a new version snapshot should be created.\npub fn should_mark_next_time_travel_version_snapshot(delta: &HummockVersionDelta) -> bool {","sourceCodeStart":838,"sourceCodeEnd":874,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/hummock/manager/time_travel.rs#L838-L874","documentation":"During time-travel archive replay, each HummockVersionDelta must extend the chain monotonically: the replay cursor (last_version) must never have moved past the delta's prev_id. If d.prev_id < last_version.id, the archive is corrupted, deltas were applied out of order, or a non-time-travel delta leaked into the archive, so replay aborts instead of producing a bogus historical version.","triggerScenarios":"replay_archive(), called from epoch_to_version and the test test_replay_archive_delta_chain, when iterating archived deltas and encountering a delta whose prev_id is older than the version already replayed.","commonSituations":"Corrupted or hand-edited time-travel archive tables; a delta archived twice or out of epoch order; compaction/time-travel delta classification bugs (the debug_assert above also guards against unexpected time travel deltas slipping in).","solutions":["Inspect the time-travel archive deltas around the failing delta id and verify they are ordered by epoch/prev_id.","Check that only NewL0SubLevel deltas are archived (should_mark_next_time_travel_version_snapshot must be false for each archived delta).","Rebuild the time-travel archive from scratch if it is corrupt; replay cannot skip or reorder deltas.","If reproducible, file a bug with the delta id, prev_id and reached version id from the message."],"exampleFix":"// before: replay assumes arbitrary archive order\nlet last_version = replay_archive(deltas);\n// after: pre-sort deltas by prev_id/epoch and drop duplicates before replay\nlet mut deltas: Vec<_> = deltas;\ndeltas.sort_by_key(|d| (d.prev_id, d.id));\ndeltas.dedup_by_key(|d| d.id);\nlet last_version = replay_archive(deltas);","handlingStrategy":"validation","validationCode":"// Rust: before replay, ensure chain monotonicity\nfn deltas_are_chained(deltas: &[HummockVersionDelta]) -> bool {\n    let mut reached = base_version_id;\n    deltas.iter().all(|d| d.prev_id >= reached && { reached = d.id; true })\n}","typeGuard":null,"tryCatchPattern":"match replay_archive(&deltas) {\n    Ok(v) => /* use v */,\n    Err(e) if e.to_string().contains(\"invalid time travel delta chain\") => /* quarantine archive, rebuild from scratch */,\n    Err(e) => return Err(e),\n}","preventionTips":["Never edit or prune time-travel archive rows out of band.","Archive deltas strictly in epoch order.","Add a debug_assert/log pipeline that flags deltas where should_mark_next_time_travel_version_snapshot is true."],"tags":["rust","meta-service","hummock","time-travel","data-corruption"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}