{"record":{"id":"9ecfb06f84fb377d","repo":"risingwavelabs/risingwave","slug":"recovered-snapshot-backfill-job-has-no-snapshot","errorCode":null,"errorMessage":"recovered snapshot backfill job {} has no snapshot backfill info","messagePattern":"recovered snapshot backfill job (.+?) has no snapshot backfill info","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src/meta/src/barrier/rpc.rs","lineNumber":825,"sourceCode":"            );\n            if committed_epoch == barrier_info.prev_epoch() {\n                info!(\n                    \"recovered creating snapshot backfill job {} catch up with upstream already\",\n                    job_id\n                );\n                database_jobs\n                    .try_insert(job_id, (fragment_infos, true))\n                    .expect(\"non-duplicate\");\n                continue;\n            }\n            let snapshot_backfill_info = StreamFragmentGraph::collect_snapshot_backfill_info_impl(\n                fragment_infos\n                    .values()\n                    .map(|fragment| (&fragment.nodes, fragment.fragment_type_mask)),\n            )?\n            .0\n            .ok_or_else(|| {\n                anyhow!(\n                    \"recovered snapshot backfill job {} has no snapshot backfill info\",\n                    job_id\n                )\n            })?;\n            let mut snapshot_epoch = None;\n            let upstream_table_ids: HashSet<_> = snapshot_backfill_info\n                .upstream_mv_table_id_to_backfill_epoch\n                .keys()\n                .cloned()\n                .collect();\n            for (upstream_table_id, epoch) in\n                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch\n            {\n                let epoch = epoch.ok_or_else(|| anyhow!(\"recovered snapshot backfill job {} to upstream {} has not set snapshot epoch\", job_id, upstream_table_id))?;\n                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);\n                if *snapshot_epoch != epoch {\n                    return Err(anyhow!(\"snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream\", epoch, upstream_table_id, snapshot_epoch).into());\n                }","sourceCodeStart":807,"sourceCodeEnd":843,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/rpc.rs#L807-L843","documentation":"Same invariant as the batch-refresh variant but on the recovery path of `inject_database_initial_barrier`: a snapshot backfill job being recovered must expose snapshot backfill info derived from its persisted fragments; when the lookup returns None the recovery aborts with this error. It indicates the recovered job's fragment metadata lacks the expected snapshot backfill node.","triggerScenarios":"Recovery/inject-initial-barrier over a snapshot backfill job whose persisted fragment infos contain no fragment with snapshot backfill type info.","commonSituations":"Meta recovery after crash/upgrade; metadata written by an older version without snapshot backfill info; corrupted or partially persisted fragment graph for the job.","solutions":["Dump the job's fragment infos from the meta store and verify a snapshot backfill fragment exists.","Restore meta state from a consistent snapshot/backup if recovery state is corrupted.","Upgrade consistently: ensure all persisted jobs were created by a version that persists snapshot backfill info.","If unrecoverable, drop and recreate the affected job."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// On recovery, pre-check persisted fragment infos before injecting the barrier\nif !fragment_infos.values().any(|f| {\n    f.nodes.values().any(|n| n.snapshot_backfill_info.is_some())\n}) {\n    // route to a repair/recreate path instead of failing injection\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Persist snapshot backfill info atomically with the job's fragment graph.","Run meta-state consistency checks during startup recovery.","Keep backups of the meta store before upgrades."],"tags":["meta","recovery","backfill","invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}