{"record":{"id":"5b87539022650865","repo":"FuelLabs/fuel-core","slug":"couldn-t-restore-from-latest-backup-for-path","errorCode":null,"errorMessage":"Couldn't restore from latest backup for path `{}`: {}","messagePattern":"Couldn't restore from latest backup for path `(.+?)`: (.+?)","errorType":"exception","errorClass":"DatabaseError::RestoreError","httpStatus":null,"severity":"critical","filePath":"crates/fuel-core/src/state/rocks_db.rs","lineNumber":863,"sourceCode":"\n    /// We delegate opening of restored db to consumer, so they can apply their own options\n    #[cfg(feature = \"backup\")]\n    pub fn restore<P: AsRef<Path> + ?Sized>(\n        db_dir: &P,\n        backup_dir: &P,\n    ) -> DatabaseResult<()> {\n        use rocksdb::backup::RestoreOptions;\n\n        let mut backup_engine = Self::backup_engine(backup_dir)?;\n        let restore_option = RestoreOptions::default();\n\n        let db_dir = db_dir.as_ref().join(Description::name());\n        let db_dir_path = db_dir.as_path();\n        // we use the default wal directory, which is same as db path\n        backup_engine\n            .restore_from_latest_backup(db_dir_path, db_dir_path, &restore_option)\n            .map_err(|e| {\n                DatabaseError::RestoreError(anyhow::anyhow!(\n                    \"Couldn't restore from latest backup for path `{}`: {}\",\n                    db_dir_path.display(),\n                    e\n                ))\n            })?;\n\n        Ok(())\n    }\n\n    pub fn shutdown(&self) {\n        // Signal rocksdb's background compaction/flush threads to stop. We\n        // use `wait=false` so this returns immediately — each `CombinedDatabase`\n        // has several layered DBs and `wait=true` per-DB serialises into a\n        // shutdown that blows past tight per-service timeouts in tests\n        // (`tests/tests/poa.rs` budgets 1s for `send_stop_signal_and_await_shutdown`).\n        // The work is still cancelled; rocksdb's own destructor finishes the\n        // teardown when the last `Arc<PrimaryInstance>` clone is dropped.\n        self.db.cancel_all_background_work(false);","sourceCodeStart":845,"sourceCodeEnd":881,"githubUrl":"https://github.com/FuelLabs/fuel-core/blob/b9d4d170da3a31c9ace5f963d633b326348e0d42/crates/fuel-core/src/state/rocks_db.rs#L845-L881","documentation":"RestoreOptions-based restore_from_latest_backup failed for the target db path, wrapped as DatabaseError::RestoreError. Restoration copies the newest backup into db_dir/Description::name(); failure typically means there is no backup to restore from, the target directory is not usable, or the latest backup itself is damaged.","triggerScenarios":"Calling restore with a backup directory that contains no backups (fresh or empty), a target directory with conflicting existing files, permission or disk problems, or a latest backup whose files were truncated by a crashed run.","commonSituations":"Disaster recovery pointed at the wrong backup directory; restoring over a live node's db directory; backups truncated by an interrupted earlier backup job.","solutions":["Confirm the backup directory actually contains backups (backup_metadata and sst files) before restoring.","Restore into an empty or removed target directory and stop the node first.","Check permissions and free disk space on both source and target volumes.","If the latest backup is corrupt, restore from an older backup id or create a fresh backup once the source is healthy."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"fn backup_exists(backup_dir: &std::path::Path, name: &str) -> bool {\n    let dir = backup_dir.join(name);\n    dir.join(\"backup_metadata\").exists()\n        || std::fs::read_dir(&dir).map(|mut d| d.next().is_some()).unwrap_or(false)\n}\n\n// before restore: require backup_exists(...) and an empty target dir","typeGuard":null,"tryCatchPattern":"match RocksDb::<Description>::restore(db_dir, backup_dir) {\n    Err(e) if e.to_string().contains(\"Couldn't restore from latest backup\") => {\n        // verify a backup exists, clear/replace the target dir, check permissions, retry;\n        // if the latest backup is corrupt, fall back to an older backup\n    }\n    rest => rest,\n}","preventionTips":["Verify backup contents (metadata plus sst files) before any restore drill.","Always restore into an empty target directory with the node stopped.","Periodically test restores; an untested backup is a hope, not a plan."],"tags":["rocksdb","backup","restore","disaster-recovery","fuel-core","rust"],"backgroundTag":null,"analyzedSha":"b9d4d170da3a31c9ace5f963d633b326348e0d42","analyzedAt":"2026-08-16T08:56:42.692Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}