{"record":{"id":"9517cde88adf2879","repo":"risingwavelabs/risingwave","slug":"recovered-snapshot-backfill-job-to-upstream","errorCode":null,"errorMessage":"recovered snapshot backfill job {} to upstream {} has not set snapshot epoch","messagePattern":"recovered snapshot backfill job (.+?) to upstream (.+?) has not set snapshot epoch","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src/meta/src/barrier/rpc.rs","lineNumber":839,"sourceCode":"                    .map(|fragment| (&fragment.nodes, fragment.fragment_type_mask)),\n            )?\n            .0\n            .ok_or_else(|| {\n                anyhow!(\n                    \"recovered snapshot backfill job {} has no snapshot backfill info\",\n                    job_id\n                )\n            })?;\n            let mut snapshot_epoch = None;\n            let upstream_table_ids: HashSet<_> = snapshot_backfill_info\n                .upstream_mv_table_id_to_backfill_epoch\n                .keys()\n                .cloned()\n                .collect();\n            for (upstream_table_id, epoch) in\n                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch\n            {\n                let epoch = epoch.ok_or_else(|| anyhow!(\"recovered snapshot backfill job {} to upstream {} has not set snapshot epoch\", job_id, upstream_table_id))?;\n                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);\n                if *snapshot_epoch != epoch {\n                    return Err(anyhow!(\"snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream\", epoch, upstream_table_id, snapshot_epoch).into());\n                }\n            }\n            let snapshot_epoch = snapshot_epoch.ok_or_else(|| {\n                anyhow!(\n                    \"snapshot backfill job {} has not set snapshot epoch\",\n                    job_id\n                )\n            })?;\n            for upstream_table_id in &upstream_table_ids {\n                subscribers\n                    .entry(*upstream_table_id)\n                    .or_default()\n                    .try_insert(job_id.as_subscriber_id(), SubscriberType::SnapshotBackfill)\n                    .expect(\"non-duplicate\");\n            }","sourceCodeStart":821,"sourceCodeEnd":857,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/rpc.rs#L821-L857","documentation":"While recovering a snapshot backfill job, `inject_database_initial_barrier` iterates `upstream_mv_table_id_to_backfill_epoch`; each upstream entry's epoch is an `Option` and must be set. `epoch.ok_or_else(...)` throws this error when some upstream MV still has no snapshot epoch recorded. It means the backfill mapping is present but incomplete.","triggerScenarios":"Recovery encountering a snapshot backfill job where at least one entry in `upstream_mv_table_id_to_backfill_epoch` maps to `None`.","commonSituations":"Job persisted mid-initialization (upstreams registered, epochs not yet assigned); crash between registering upstreams and setting epochs; state written by a buggy or older code path.","solutions":["Recover from an earlier consistent meta snapshot taken before the partial write.","Recreate the snapshot backfill job so all upstream epochs are assigned atomically.","Check meta logs to find which upstream table was incomplete and inspect the writer path.","Report as a bug if a completed job shows unset epochs after clean recovery."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify every upstream epoch is set before attempting recovery\nlet all_set = snapshot_backfill_info\n    .upstream_mv_table_id_to_backfill_epoch\n    .values()\n    .all(|e| e.is_some());\nif !all_set {\n    // repair or recreate the job before injection\n}","typeGuard":"// Rust: Option<Epoch> narrowing\nfn set_epochs(m: &HashMap<TableId, Option<Epoch>>) -> Option<HashMap<TableId, Epoch>> {\n    m.iter().map(|(k, v)| Some((*k, v?))).collect()\n}","tryCatchPattern":null,"preventionTips":["Assign upstream epochs atomically when building snapshot backfill info.","Never persist a job whose upstream map contains None epochs.","Alert on jobs whose backfill epochs stay unset beyond initialization."],"tags":["meta","recovery","backfill","epoch"],"backgroundTag":"missing-required-config-field","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}