FuelLabs/fuel-core · error · DatabaseError::BackupEngineInitError

Couldn't create environment for backup: {}

Error message

Couldn't create environment for backup: {}

What it means

Creating the RocksDB Env (environment and thread-pool handle) for the backup engine failed, wrapped as DatabaseError::BackupEngineInitError. Env::new is a thin wrapper over OS resource creation, so failures are rare and system-level: file-descriptor exhaustion, inability to spawn threads, or severe memory pressure.

Source

Thrown at crates/fuel-core/src/state/rocks_db.rs:795

        let backup_dir = backup_dir.as_ref().join(Description::name());
        let backup_dir_path = backup_dir.as_path();

        let mut backup_engine_options = BackupEngineOptions::new(backup_dir_path)
            .map_err(|e| {
                DatabaseError::BackupEngineInitError(anyhow::anyhow!(
                    "Couldn't create backup engine options for path `{}`: {}",
                    backup_dir_path.display(),
                    e
                ))
            })?;

        let cpu_number =
            i32::try_from(num_cpus::get()).expect("The number of CPU can't exceed `i32`");

        backup_engine_options.set_max_background_operations(cmp::max(1, cpu_number / 4));

        let env = Env::new().map_err(|e| {
            DatabaseError::BackupEngineInitError(anyhow::anyhow!(
                "Couldn't create environment for backup: {}",
                e
            ))
        })?;

        let backup_engine =
            BackupEngine::open(&backup_engine_options, &env).map_err(|e| {
                DatabaseError::BackupEngineInitError(anyhow::anyhow!(
                    "Couldn't open backup engine for path `{}`: {}",
                    backup_dir.display(),
                    e
                ))
            })?;

        Ok(backup_engine)
    }

    #[cfg(feature = "backup")]

View on GitHub (pinned to b9d4d170da)

Solutions

  1. Raise file-descriptor and process limits (ulimit -n, systemd LimitNOFILE, container pids/memory limits).
  2. Reduce load or memory pressure before running the backup, including lowering rocksdb background thread counts.
  3. Retry once pressure is relieved; if it persists, inspect the trailing error plus dmesg or OOM logs.
Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-flight resource sanity check (approximate)
fn resources_sane() -> bool {
    let fds = std::fs::read_to_string("/proc/self/limits").map(|s| {
        s.lines().any(|l| l.starts_with("Max open files") && !l.contains("unlimited"))
    });
    let threads = std::thread::available_parallelism().map(|n| n.get() > 0).unwrap_or(false);
    fds.is_ok() && threads
}

Try / catch

match RocksDb::<Description>::backup(db_dir, backup_dir, db_config) {
    Err(e) if e.to_string().contains("Couldn't create environment for backup") => {
        // raise ulimits / container limits, reduce load, then retry once
    }
    rest => rest,
}

Prevention

When it happens

Trigger: Running backup while the process is near OS limits — out of file descriptors (ulimit -n), thread creation blocked by a pids cgroup limit, or heavy memory pressure.

Common situations: Backups scheduled on heavily loaded nodes with many RocksDB background threads; restrictive container limits; small default ulimits under systemd service managers.

Related errors

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/2bae8017e66cf20c. Report an issue: GitHub.