risingwavelabs/risingwave · critical · BackupError

referenced objects not found in object store: {:?}

Error message

referenced objects not found in object store: {:?}

What it means

During a restore, `validate_integrity` verifies that every object ID referenced by the backup metadata actually exists in the target object store. If any referenced objects are absent after checking the store, it fails with the list of missing object IDs, preventing an incomplete restore that would later hit unresolvable manifest/SST references.

Source

Thrown at src/meta/src/backup_restore/restore.rs:310

            Arc::new(ObjectStoreConfig::default()),
        )
        .await,
    );
    let mut iter = object_store
        .list(hummock_storage_directory, None, None)
        .await?;
    while let Some(obj) = iter.try_next().await? {
        let Some(obj_id) = try_get_object_id_from_path(&obj.key) else {
            continue;
        };
        if object_ids.remove(&obj_id.as_raw()) && object_ids.is_empty() {
            break;
        }
    }
    if object_ids.is_empty() {
        return Ok(());
    }
    Err(BackupError::Other(anyhow!(
        "referenced objects not found in object store: {:?}",
        object_ids
    )))
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check the restore config (bucket, endpoint, prefix) points at the object store the backup was actually written to.
2. Compare the reported object_ids against the backup's metadata to confirm whether the backup itself is incomplete.
3. Re-create the backup from a healthy cluster if objects are permanently missing; a partial backup cannot be repaired in place.
4. Check whether a retention/vacuum job deleted objects and restore from an earlier intact backup.

Example fix

// before: restore reads from default/wrong store
store = create_backup_store(cfg.backup_storage_url).await?;
// after: point at the store the backup was taken from
store = create_backup_store("s3://correct-bucket/rw-backup").await?;

// before restore: verify all referenced objects exist
let missing: Vec<_> = stream::iter(object_ids.iter())
    .filter_map(|id| async move {
        match store.head_object(*id).await { Ok(_) => None, Err(_) => Some(*id) }
    })
    .collect().await;
if !missing.is_empty() { return Err(format!("missing objects: {missing:?}")); }

When it happens

Trigger: Running a restore (via restore_impl) whose backup metadata references objects (SSTs, manifests, metadata snapshots) that are missing from the target object store — e.g. objects deleted or never uploaded, or restore pointed at the wrong bucket/prefix.

Common situations: Restoring against the wrong S3 bucket/endpoint config; backup was cleaned up by retention policy (vacuum) while metadata remains; partial backup upload due to earlier failure; restoring a backup copied incompletely between clusters.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Read backup error: {0}
• backup job failed: job {}, {}
• snapshot id {} not found
• {k} not found in system_parameter table
• Meta storage is not empty before being restored

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/cb5595c45eba632d. Report an issue: GitHub.