{"record":{"id":"fb8bad558fb31404","repo":"risingwavelabs/risingwave","slug":"backup-job-failed-job","errorCode":null,"errorMessage":"backup job failed: job {}, {}","messagePattern":"backup job failed: job (.+?), (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/ctl/src/cmd_impl/meta/backup_meta.rs","lineNumber":45,"sourceCode":"        match job_status {\n            BackupJobStatus::Running => {\n                tracing::info!(\"backup job is still running: job {}, {}\", job_id, message);\n                tokio::time::sleep(Duration::from_secs(1)).await;\n            }\n            BackupJobStatus::Succeeded => {\n                tracing::info!(\"backup job succeeded: job {}, {}\", job_id, message);\n                tracing::info!(\"rw version: {}\", RW_VERSION);\n                break;\n            }\n            BackupJobStatus::NotFound => {\n                return Err(anyhow::anyhow!(\n                    \"backup job status not found: job {}, {}\",\n                    job_id,\n                    message\n                ));\n            }\n            BackupJobStatus::Failed => {\n                return Err(anyhow::anyhow!(\n                    \"backup job failed: job {}, {}\",\n                    job_id,\n                    message\n                ));\n            }\n            _ => unreachable!(\"unknown backup job status\"),\n        }\n    }\n    Ok(())\n}\n\npub async fn delete_meta_snapshots(\n    context: &CtlContext,\n    snapshot_ids: &[u64],\n) -> anyhow::Result<()> {\n    let meta_client = context.meta_client().await?;\n    meta_client.delete_meta_snapshot(snapshot_ids).await?;\n    tracing::info!(\"delete meta snapshots succeeded: {:?}\", snapshot_ids);","sourceCodeStart":27,"sourceCodeEnd":63,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/ctl/src/cmd_impl/meta/backup_meta.rs#L27-L63","documentation":"While polling a meta backup job, the command received an explicit Failed status from the meta node and surfaces the job id plus the meta-provided failure message. The backup did not complete and no metadata snapshot was taken.","triggerScenarios":"The meta-side backup job failed — e.g. failure writing the snapshot to the object store, invalid meta store state, or an internal error reported by the meta node during `rw meta backup-meta`.","commonSituations":"Object store credentials/bucket misconfigured so the meta node cannot write the backup; storage quota exceeded; concurrent backup or maintenance on meta.","solutions":["Read the meta node logs for the same job id to find the underlying failure cause","Verify the object store config (credentials, bucket, permissions) used by the meta node","Free up space or fix quotas if the backup failed on write","Retry the backup after fixing the root cause"],"exampleFix":"null","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if let Err(e) = backup_meta(ctx).await {\n    eprintln!(\"backup failed: {e:#}; check meta logs for the job id\");\n    std::process::exit(1);\n}","preventionTips":["Verify object store credentials/bucket config on the meta node before backing up","Monitor meta logs during the job to catch early failures","Ensure sufficient storage quota for the snapshot"],"tags":["meta","backup","job-failure","object-store"],"backgroundTag":"job-execution-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}