What causes "version delta {} not found" in risingwavelabs/risingwave?

Verify meta store consistency: check hummock_version_delta rows for the reported delta_id.

risingwavelabs/risingwave · error · TimeTravel

version delta {} not found

Error message

version delta {} not found

What it means

While truncating time-travel metadata, each version delta ID scheduled for deletion must be present in the set of deltas loaded in the same transaction. If a delta ID to delete is missing from the in-transaction map (delta_to_delete_by_id), the code returns Error::TimeTravel("version delta {} not found"), an internal invariant violation indicating the metadata table is inconsistent with the deletion plan.

Source

Thrown at src/meta/src/hummock/manager/time_travel.rs:314

            Ok(())
        }
        for delta_id_batch in delta_ids_to_delete.chunks(delta_fetch_batch_size) {
            let mut delta_to_delete_by_id: HashMap<_, _> =
                hummock_time_travel_delta::Entity::find()
                    .filter(
                        hummock_time_travel_delta::Column::VersionId
                            .is_in(delta_id_batch.iter().copied()),
                    )
                    .all(&txn)
                    .await?
                    .into_iter()
                    .map(|delta| (delta.version_id, delta))
                    .collect();
            for &delta_id_to_delete in delta_id_batch {
                let delta_to_delete = delta_to_delete_by_id
                    .remove(&delta_id_to_delete)
                    .ok_or_else(|| {
                        Error::TimeTravel(anyhow!(format!(
                            "version delta {} not found",
                            delta_id_to_delete
                        )))
                    })?;
                let delta_to_delete = IncompleteHummockVersionDelta::from_persisted_protobuf_owned(
                    delta_to_delete.version_delta.to_protobuf(),
                );
                let new_sst_ids = delta_to_delete.newly_added_sst_ids();
                // The SST ids added and then deleted by compaction between the 2 versions.
                sst_ids_to_delete.extend(&new_sst_ids - &retained_snapshot_sst_ids);
                if sst_ids_to_delete.len() >= delete_sst_batch_size {
                    delete_sst_in_batch(
                        &txn,
                        std::mem::take(&mut sst_ids_to_delete),
                        delete_sst_batch_size,
                    )
                    .await?;
                }

View on GitHub (pinned to 6469eb736d)

Solutions

Verify meta store consistency: check hummock_version_delta rows for the reported delta_id.
Ensure no concurrent delete_time_travel_metadata / GC tasks run simultaneously; serialize version GC.
If metadata is inconsistent, restore the meta store from backup or rebuild time-travel metadata.
Capture meta node logs and report the error, as it usually indicates an internal bug (file an issue with the delta_id).

Defensive patterns

Strategy: try-catch

Validate before calling

// confirm the delta rows still exist before deleting
let missing: Vec<_> = delta_ids.iter()
    .filter(|id| !existing_delta_ids.contains(id))
    .collect();
if !missing.is_empty() { /* abort plan: metadata inconsistent */ }

Try / catch

// match the typed TimeTravel error
match meta.delete_time_travel_metadata(...).await {
    Err(Error::TimeTravel(e)) if e.to_string().contains("not found") => {
        // mark metadata inconsistent; stop GC, alert, consider restore
    }
    other => other,
}

Prevention

Run a single serialized version-GC worker; never truncate time-travel metadata concurrently.
Never manually delete rows from hummock_version_delta.
Back up the meta store before version GC operations.
Alert on any TimeTravel internal errors — they indicate inconsistent metadata.

When it happens

Trigger: Calling delete_time_travel_metadata (via version GC / hummock admin) when a hummock_version_delta row referenced by the deletion batch has already been deleted or never existed — typically from concurrent metadata truncation, corrupted meta store state, or a partial prior cleanup.

Common situations: Concurrent time-travel GC tasks racing; a previous failed truncation left inconsistent state; manual cleanup of hummock_version_delta rows in the meta store DB.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/fc7c052b631ae027. Report an issue: GitHub.