risingwavelabs/risingwave · error · BackupError

concurrent backup job is not supported: existent job {}

Error message

concurrent backup job is not supported: existent job {}

What it means

BackupManager allows only one meta backup job at a time; `start_backup_job` checks `running_job_handle` under a lock and bails if a job is already registered. The error includes the id of the currently running job.

Source

Thrown at src/meta/src/backup_restore/backup_manager.rs:203

            env,
            hummock_manager,
            Arc::new(MetaMetrics::default()),
            (
                risingwave_backup::storage::unused().await,
                StoreConfig::default(),
            ),
        )
    }

    /// Starts a backup job in background. It's non-blocking.
    /// Returns job id.
    pub async fn start_backup_job(
        self: &Arc<Self>,
        remarks: Option<String>,
    ) -> MetaResult<MetaBackupJobId> {
        let mut guard = self.running_job_handle.lock().await;
        if let Some(job) = (*guard).as_ref() {
            bail!(format!(
                "concurrent backup job is not supported: existent job {}",
                job.job_id
            ));
        }
        // The reasons to limit number of meta snapshot are:
        // 1. limit size of `MetaSnapshotManifest`, which is kept in memory by
        // `ObjectStoreMetaSnapshotStorage`.
        // 2. limit number of pinned SSTs returned by
        // `list_pinned_ssts`, which subsequently is used by GC.
        const MAX_META_SNAPSHOT_NUM: usize = 100;
        let current_number = self
            .backup_store
            .load()
            .0
            .manifest()
            .await
            .snapshot_metadata
            .len();

View on GitHub (pinned to 6469eb736d)

Solutions

1. Wait for the existing job (id in the message) to complete before starting a new one
2. If the previous job is dead/stale, restart the meta node or clear the running job handle so a new backup can start
3. Add scheduling/locking in tooling to prevent concurrent backup invocations

Example fix

// before
let id = backup_manager.start_backup_job(None).await?; // may conflict
// after
if let Some(job) = backup_manager.get_running_job().await { /* wait or abort */ }
let id = backup_manager.start_backup_job(None).await?;

if backup_manager.get_running_job().await.is_some() { /* defer or abort */ }

match backup_manager.start_backup_job(remarks).await {
    Err(e) if e.to_string().contains("concurrent backup job") => { /* wait for running job */ }
    other => other?,
}

When it happens

Trigger: Calling start_backup_job (e.g. via `risectl meta backup` or internal recovery flows) while another backup job is still running and has not been finished or cleaned up.

Common situations: Operator triggers backup twice in quick succession; a previous backup crashed without clearing the running-job handle; automated scripts with overlapping schedules.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• too many existent meta snapshots, expect at most {}
• inconsistent hummock version: expected {}, actual {}
• snapshot id {} not found
• backup job status not found: job {}, {}
• backup job failed: job {}, {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/0c86d125d66b36c3. Report an issue: GitHub.