{"record":{"id":"35e63c5391e068aa","repo":"risingwavelabs/risingwave","slug":"snapshot-epoch-to-upstream-different-to-snap","errorCode":null,"errorMessage":"snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream","messagePattern":"snapshot epoch (.+?) to upstream (.+?) different to snapshot epoch (.+?) to previous upstream","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src/meta/src/barrier/rpc.rs","lineNumber":842,"sourceCode":"            .ok_or_else(|| {\n                anyhow!(\n                    \"recovered snapshot backfill job {} has no snapshot backfill info\",\n                    job_id\n                )\n            })?;\n            let mut snapshot_epoch = None;\n            let upstream_table_ids: HashSet<_> = snapshot_backfill_info\n                .upstream_mv_table_id_to_backfill_epoch\n                .keys()\n                .cloned()\n                .collect();\n            for (upstream_table_id, epoch) in\n                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch\n            {\n                let epoch = epoch.ok_or_else(|| anyhow!(\"recovered snapshot backfill job {} to upstream {} has not set snapshot epoch\", job_id, upstream_table_id))?;\n                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);\n                if *snapshot_epoch != epoch {\n                    return Err(anyhow!(\"snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream\", epoch, upstream_table_id, snapshot_epoch).into());\n                }\n            }\n            let snapshot_epoch = snapshot_epoch.ok_or_else(|| {\n                anyhow!(\n                    \"snapshot backfill job {} has not set snapshot epoch\",\n                    job_id\n                )\n            })?;\n            for upstream_table_id in &upstream_table_ids {\n                subscribers\n                    .entry(*upstream_table_id)\n                    .or_default()\n                    .try_insert(job_id.as_subscriber_id(), SubscriberType::SnapshotBackfill)\n                    .expect(\"non-duplicate\");\n            }\n            ongoing_snapshot_backfill_jobs\n                .try_insert(\n                    job_id,","sourceCodeStart":824,"sourceCodeEnd":860,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/rpc.rs#L824-L860","documentation":"During recovery, all upstream MVs of a snapshot backfill job must agree on a single snapshot epoch. The code keeps the first seen epoch via `get_or_insert` and throws this error when a later upstream's epoch differs from the accumulated one. Mixed epochs mean the job's upstreams were snapshotted inconsistently and recovery cannot proceed.","triggerScenarios":"`inject_database_initial_barrier` recovering a snapshot backfill job whose `upstream_mv_table_id_to_backfill_epoch` contains two different epoch values across upstream tables.","commonSituations":"Partial/concurrent upstream registration across different barrier epochs; metadata corruption; jobs mutated by mismatched code versions.","solutions":["Inspect the per-upstream epochs in the job's snapshot backfill info to identify the inconsistent upstream.","Recreate the snapshot backfill job so all upstreams are captured in one barrier/epoch.","Restore a consistent meta snapshot from before the divergence.","File a bug with the epoch values and job_id if this appears on an untouched job."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Confirm all upstreams share one epoch before recovery\nlet epochs: HashSet<_> = snapshot_backfill_info\n    .upstream_mv_table_id_to_backfill_epoch\n    .values()\n    .flatten()\n    .copied()\n    .collect();\nassert!(epochs.len() <= 1, \"inconsistent snapshot epochs: {epochs:?}\");","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Capture all upstream MVs within a single barrier so they share one snapshot epoch.","Add an invariant check when persisting snapshot backfill info.","Log per-upstream epochs to make divergence diagnosable."],"tags":["meta","recovery","backfill","epoch"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}