{"record":{"id":"f16c5768bcd8aade","repo":"risingwavelabs/risingwave","slug":"should-not-pause-when-having-snapshot-backfill-job","errorCode":null,"errorMessage":"should not pause when having snapshot backfill job {job_id}","messagePattern":"should not pause when having snapshot backfill job (.+?)","errorType":"error_code","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src/meta/src/barrier/rpc.rs","lineNumber":1074,"sourceCode":"                    &fragment_infos.nodes,\n                    fragment_infos.actors.iter().map(move |(actor_id, actor)| {\n                        (\n                            stream_actors.get(actor_id).expect(\"should exist\"),\n                            actor.worker_id,\n                        )\n                    }),\n                    vec![], // no subscribers for backfilling jobs,\n                )\n            }));\n\n            let database_job_source_splits =\n                collect_source_splits(database_jobs.values().flatten(), source_splits);\n            assert!(\n                !cdc_table_snapshot_splits.contains_key(&job_id),\n                \"snapshot backfill job {job_id} should not have cdc backfill\"\n            );\n            if is_paused {\n                bail!(\"should not pause when having snapshot backfill job {job_id}\");\n            }\n            let job_backfill_orders = job_backfill_orders(job_extra_info, job_id);\n            let job_backfill_orders =\n                StreamFragmentGraph::extend_fragment_backfill_ordering_with_locality_backfill(\n                    job_backfill_orders,\n                    fragment_relations,\n                    || {\n                        info.iter().map(|(fragment_id, fragment)| {\n                            (*fragment_id, fragment.fragment_type_mask, &fragment.nodes)\n                        })\n                    },\n                );\n            let mutation = build_mutation(\n                &database_job_source_splits,\n                Default::default(), // no cdc backfill job for\n                &job_backfill_orders,\n                false,\n            );","sourceCodeStart":1056,"sourceCodeEnd":1092,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/rpc.rs#L1056-L1092","documentation":"`inject_database_initial_barrier` injects an initial barrier for a database-level snapshot backfill, which requires the system to be unpaused so the barrier can be collected. If the database is currently paused (`is_paused`), `bail!` aborts with this message: pausing while a snapshot backfill job exists is not supported at this point.","triggerScenarios":"Injecting the database initial barrier while `is_paused` is true for a database containing snapshot backfill job {job_id} (and after confirming that job has no CDC backfill splits).","commonSituations":"Operator paused the cluster/database (e.g. via ALTER DATABASE ... PAUSE or during maintenance) and then triggered a snapshot backfill / database recovery; automation that pauses during upgrades while backfill jobs are in flight.","solutions":["Resume the database first (e.g. `ALTER DATABASE ... RESUME`) so it is unpaused, then re-run the injection/recovery.","Defer pausing until the snapshot backfill job completes.","Rework automation/upgrade scripts to check for active snapshot backfill jobs before pausing.","If pausing is essential, cancel/recreate the backfill job per supported procedure instead of pausing mid-backfill."],"exampleFix":"// before\nALTER DATABASE db PAUSE;\n-- trigger snapshot backfill\n// after\nALTER DATABASE db RESUME;\n-- wait for snapshot backfill job to finish\nALTER DATABASE db PAUSE;","handlingStrategy":"validation","validationCode":"// SQL-level pre-check before triggering recovery/backfill\n-- SELECT ... must show the database is not paused\n-- and no active snapshot backfill jobs exist\nCHECK: run `SHOW VARIABLES` / cluster state to confirm is_paused = false\nbefore calling inject_database_initial_barrier.","typeGuard":null,"tryCatchPattern":"match inject_database_initial_barrier(...).await {\n    Err(e) if e.to_string().contains(\"should not pause when having snapshot backfill job\") => {\n        // resume database, then retry\n        resume_database(db_id).await?;\n        inject_database_initial_barrier(...).await?;\n    }\n    other => other?,\n}","preventionTips":["Check paused state before running database-level snapshot backfill or recovery.","Sequence pause/resume automation after backfill jobs complete.","Add a guard in ops tooling that blocks PAUSE while snapshot backfill jobs are active."],"tags":["meta","barrier","pause","backfill"],"backgroundTag":"invalid-state-transition","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}