What causes "backup job status not found: job {}, {}" in risingwavelabs/risingwave?

Re-run the backup-meta command to create a fresh backup job

backup job status not found: job {}, {}

Error message

backup job status not found: job {}, {}

What it means

backup_meta polls the meta node for a backup job's status. If the polled status comes back NotFound, the job (or its status) does not exist on the meta side, so the ctl command returns this error including the job id and message.

Source

Thrown at src/ctl/src/cmd_impl/meta/backup_meta.rs:38

use crate::CtlContext;

pub async fn backup_meta(context: &CtlContext, remarks: Option<String>) -> anyhow::Result<()> {
    let meta_client = context.meta_client().await?;
    let job_id = meta_client.backup_meta(remarks).await?;
    loop {
        let (job_status, message) = meta_client.get_backup_job_status(job_id).await?;
        match job_status {
            BackupJobStatus::Running => {
                tracing::info!("backup job is still running: job {}, {}", job_id, message);
                tokio::time::sleep(Duration::from_secs(1)).await;
            }
            BackupJobStatus::Succeeded => {
                tracing::info!("backup job succeeded: job {}, {}", job_id, message);
                tracing::info!("rw version: {}", RW_VERSION);
                break;
            }
            BackupJobStatus::NotFound => {
                return Err(anyhow::anyhow!(
                    "backup job status not found: job {}, {}",
                    job_id,
                    message
                ));
            }
            BackupJobStatus::Failed => {
                return Err(anyhow::anyhow!(
                    "backup job failed: job {}, {}",
                    job_id,
                    message
                ));
            }
            _ => unreachable!("unknown backup job status"),
        }
    }
    Ok(())
}

View on GitHub (pinned to 6469eb736d)

Solutions

Re-run the backup-meta command to create a fresh backup job
Verify the meta node is the same leader that created the job (no failover in between)
Check meta logs around the job id to confirm whether it was created and garbage-collected

Example fix

null

Defensive patterns

Strategy: try-catch

Try / catch

match res {
    Err(e) if e.to_string().contains("backup job status not found") => {
        eprintln!("job lost (meta restart?); re-run backup-meta");
    }
    other => other.expect("backup failed"),
}

Prevention

Use the job id immediately after creating it; never reuse old ids
Avoid meta failovers/restarts during backups
Confirm meta leader stability before long-running backup jobs

When it happens

Trigger: Polling a backup job id that was never created, was already cleaned up, or whose record was lost after a meta restart/failover during `rw meta backup-meta`.

Common situations: Meta node restarted mid-backup losing the job record; querying with a stale job id from a previous cluster; a bug/race where the status is read before the job is registered.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/ad5cac5a93d1783e. Report an issue: GitHub.