risingwavelabs/risingwave · error

backup job failed: job {}, {}

Error message

backup job failed: job {}, {}

What it means

While polling a meta backup job, the command received an explicit Failed status from the meta node and surfaces the job id plus the meta-provided failure message. The backup did not complete and no metadata snapshot was taken.

Source

Thrown at src/ctl/src/cmd_impl/meta/backup_meta.rs:45

        match job_status {
            BackupJobStatus::Running => {
                tracing::info!("backup job is still running: job {}, {}", job_id, message);
                tokio::time::sleep(Duration::from_secs(1)).await;
            }
            BackupJobStatus::Succeeded => {
                tracing::info!("backup job succeeded: job {}, {}", job_id, message);
                tracing::info!("rw version: {}", RW_VERSION);
                break;
            }
            BackupJobStatus::NotFound => {
                return Err(anyhow::anyhow!(
                    "backup job status not found: job {}, {}",
                    job_id,
                    message
                ));
            }
            BackupJobStatus::Failed => {
                return Err(anyhow::anyhow!(
                    "backup job failed: job {}, {}",
                    job_id,
                    message
                ));
            }
            _ => unreachable!("unknown backup job status"),
        }
    }
    Ok(())
}

pub async fn delete_meta_snapshots(
    context: &CtlContext,
    snapshot_ids: &[u64],
) -> anyhow::Result<()> {
    let meta_client = context.meta_client().await?;
    meta_client.delete_meta_snapshot(snapshot_ids).await?;
    tracing::info!("delete meta snapshots succeeded: {:?}", snapshot_ids);

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the meta node logs for the same job id to find the underlying failure cause
2. Verify the object store config (credentials, bucket, permissions) used by the meta node
3. Free up space or fix quotas if the backup failed on write
4. Retry the backup after fixing the root cause

Example fix

null

if let Err(e) = backup_meta(ctx).await {
    eprintln!("backup failed: {e:#}; check meta logs for the job id");
    std::process::exit(1);
}

When it happens

Trigger: The meta-side backup job failed — e.g. failure writing the snapshot to the object store, invalid meta store state, or an internal error reported by the meta node during `rw meta backup-meta`.

Common situations: Object store credentials/bucket misconfigured so the meta node cannot write the backup; storage quota exceeded; concurrent backup or maintenance on meta.

Related errors

• backup job status not found: job {}, {}
• concurrent backup job is not supported: existent job {}
• too many existent meta snapshots, expect at most {}
• inconsistent hummock version: expected {}, actual {}
• snapshot id {} not found

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/fb8bad558fb31404. Report an issue: GitHub.