{"record":{"id":"cb5595c45eba632d","repo":"risingwavelabs/risingwave","slug":"referenced-objects-not-found-in-object-store","errorCode":null,"errorMessage":"referenced objects not found in object store: {:?}","messagePattern":"referenced objects not found in object store: (.+?)","errorType":"exception","errorClass":"BackupError","httpStatus":null,"severity":"critical","filePath":"src/meta/src/backup_restore/restore.rs","lineNumber":310,"sourceCode":"            Arc::new(ObjectStoreConfig::default()),\n        )\n        .await,\n    );\n    let mut iter = object_store\n        .list(hummock_storage_directory, None, None)\n        .await?;\n    while let Some(obj) = iter.try_next().await? {\n        let Some(obj_id) = try_get_object_id_from_path(&obj.key) else {\n            continue;\n        };\n        if object_ids.remove(&obj_id.as_raw()) && object_ids.is_empty() {\n            break;\n        }\n    }\n    if object_ids.is_empty() {\n        return Ok(());\n    }\n    Err(BackupError::Other(anyhow!(\n        \"referenced objects not found in object store: {:?}\",\n        object_ids\n    )))\n}\n","sourceCodeStart":292,"sourceCodeEnd":315,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/backup_restore/restore.rs#L292-L315","documentation":"During a restore, `validate_integrity` verifies that every object ID referenced by the backup metadata actually exists in the target object store. If any referenced objects are absent after checking the store, it fails with the list of missing object IDs, preventing an incomplete restore that would later hit unresolvable manifest/SST references.","triggerScenarios":"Running a restore (via restore_impl) whose backup metadata references objects (SSTs, manifests, metadata snapshots) that are missing from the target object store — e.g. objects deleted or never uploaded, or restore pointed at the wrong bucket/prefix.","commonSituations":"Restoring against the wrong S3 bucket/endpoint config; backup was cleaned up by retention policy (vacuum) while metadata remains; partial backup upload due to earlier failure; restoring a backup copied incompletely between clusters.","solutions":["Check the restore config (bucket, endpoint, prefix) points at the object store the backup was actually written to.","Compare the reported object_ids against the backup's metadata to confirm whether the backup itself is incomplete.","Re-create the backup from a healthy cluster if objects are permanently missing; a partial backup cannot be repaired in place.","Check whether a retention/vacuum job deleted objects and restore from an earlier intact backup."],"exampleFix":"// before: restore reads from default/wrong store\nstore = create_backup_store(cfg.backup_storage_url).await?;\n// after: point at the store the backup was taken from\nstore = create_backup_store(\"s3://correct-bucket/rw-backup\").await?;","handlingStrategy":"validation","validationCode":"// before restore: verify all referenced objects exist\nlet missing: Vec<_> = stream::iter(object_ids.iter())\n    .filter_map(|id| async move {\n        match store.head_object(*id).await { Ok(_) => None, Err(_) => Some(*id) }\n    })\n    .collect().await;\nif !missing.is_empty() { return Err(format!(\"missing objects: {missing:?}\")); }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pin restore config to the exact backup store (bucket/prefix) used by the backup.","Disable retention/vacuum cleanup until restore completes.","Verify backup completeness (object counts vs metadata) right after taking it."],"tags":["backup","restore","object-store","data-integrity"],"backgroundTag":"resource-not-found","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}