FuelLabs/fuel-core · critical · DatabaseError::RestoreError

Couldn't restore from latest backup for path `{}`: {}

Error message

Couldn't restore from latest backup for path `{}`: {}

What it means

RestoreOptions-based restore_from_latest_backup failed for the target db path, wrapped as DatabaseError::RestoreError. Restoration copies the newest backup into db_dir/Description::name(); failure typically means there is no backup to restore from, the target directory is not usable, or the latest backup itself is damaged.

Source

Thrown at crates/fuel-core/src/state/rocks_db.rs:863

    /// We delegate opening of restored db to consumer, so they can apply their own options
    #[cfg(feature = "backup")]
    pub fn restore<P: AsRef<Path> + ?Sized>(
        db_dir: &P,
        backup_dir: &P,
    ) -> DatabaseResult<()> {
        use rocksdb::backup::RestoreOptions;

        let mut backup_engine = Self::backup_engine(backup_dir)?;
        let restore_option = RestoreOptions::default();

        let db_dir = db_dir.as_ref().join(Description::name());
        let db_dir_path = db_dir.as_path();
        // we use the default wal directory, which is same as db path
        backup_engine
            .restore_from_latest_backup(db_dir_path, db_dir_path, &restore_option)
            .map_err(|e| {
                DatabaseError::RestoreError(anyhow::anyhow!(
                    "Couldn't restore from latest backup for path `{}`: {}",
                    db_dir_path.display(),
                    e
                ))
            })?;

        Ok(())
    }

    pub fn shutdown(&self) {
        // Signal rocksdb's background compaction/flush threads to stop. We
        // use `wait=false` so this returns immediately — each `CombinedDatabase`
        // has several layered DBs and `wait=true` per-DB serialises into a
        // shutdown that blows past tight per-service timeouts in tests
        // (`tests/tests/poa.rs` budgets 1s for `send_stop_signal_and_await_shutdown`).
        // The work is still cancelled; rocksdb's own destructor finishes the
        // teardown when the last `Arc<PrimaryInstance>` clone is dropped.
        self.db.cancel_all_background_work(false);

View on GitHub (pinned to b9d4d170da)

Solutions

  1. Confirm the backup directory actually contains backups (backup_metadata and sst files) before restoring.
  2. Restore into an empty or removed target directory and stop the node first.
  3. Check permissions and free disk space on both source and target volumes.
  4. If the latest backup is corrupt, restore from an older backup id or create a fresh backup once the source is healthy.
Defensive patterns

Strategy: validation

Validate before calling

fn backup_exists(backup_dir: &std::path::Path, name: &str) -> bool {
    let dir = backup_dir.join(name);
    dir.join("backup_metadata").exists()
        || std::fs::read_dir(&dir).map(|mut d| d.next().is_some()).unwrap_or(false)
}

// before restore: require backup_exists(...) and an empty target dir

Try / catch

match RocksDb::<Description>::restore(db_dir, backup_dir) {
    Err(e) if e.to_string().contains("Couldn't restore from latest backup") => {
        // verify a backup exists, clear/replace the target dir, check permissions, retry;
        // if the latest backup is corrupt, fall back to an older backup
    }
    rest => rest,
}

Prevention

When it happens

Trigger: Calling restore with a backup directory that contains no backups (fresh or empty), a target directory with conflicting existing files, permission or disk problems, or a latest backup whose files were truncated by a crashed run.

Common situations: Disaster recovery pointed at the wrong backup directory; restoring over a live node's db directory; backups truncated by an interrupted earlier backup job.

Related errors

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/5b87539022650865. Report an issue: GitHub.