risingwavelabs/risingwave · error · anyhow::Error

should not pause when having snapshot backfill job {job_id}

Error message

should not pause when having snapshot backfill job {job_id}

What it means

`inject_database_initial_barrier` injects an initial barrier for a database-level snapshot backfill, which requires the system to be unpaused so the barrier can be collected. If the database is currently paused (`is_paused`), `bail!` aborts with this message: pausing while a snapshot backfill job exists is not supported at this point.

Source

Thrown at src/meta/src/barrier/rpc.rs:1074

                    &fragment_infos.nodes,
                    fragment_infos.actors.iter().map(move |(actor_id, actor)| {
                        (
                            stream_actors.get(actor_id).expect("should exist"),
                            actor.worker_id,
                        )
                    }),
                    vec![], // no subscribers for backfilling jobs,
                )
            }));

            let database_job_source_splits =
                collect_source_splits(database_jobs.values().flatten(), source_splits);
            assert!(
                !cdc_table_snapshot_splits.contains_key(&job_id),
                "snapshot backfill job {job_id} should not have cdc backfill"
            );
            if is_paused {
                bail!("should not pause when having snapshot backfill job {job_id}");
            }
            let job_backfill_orders = job_backfill_orders(job_extra_info, job_id);
            let job_backfill_orders =
                StreamFragmentGraph::extend_fragment_backfill_ordering_with_locality_backfill(
                    job_backfill_orders,
                    fragment_relations,
                    || {
                        info.iter().map(|(fragment_id, fragment)| {
                            (*fragment_id, fragment.fragment_type_mask, &fragment.nodes)
                        })
                    },
                );
            let mutation = build_mutation(
                &database_job_source_splits,
                Default::default(), // no cdc backfill job for
                &job_backfill_orders,
                false,
            );

View on GitHub (pinned to 6469eb736d)

Solutions

1. Resume the database first (e.g. `ALTER DATABASE ... RESUME`) so it is unpaused, then re-run the injection/recovery.
2. Defer pausing until the snapshot backfill job completes.
3. Rework automation/upgrade scripts to check for active snapshot backfill jobs before pausing.
4. If pausing is essential, cancel/recreate the backfill job per supported procedure instead of pausing mid-backfill.

Example fix

// before
ALTER DATABASE db PAUSE;
-- trigger snapshot backfill
// after
ALTER DATABASE db RESUME;
-- wait for snapshot backfill job to finish
ALTER DATABASE db PAUSE;

// SQL-level pre-check before triggering recovery/backfill
-- SELECT ... must show the database is not paused
-- and no active snapshot backfill jobs exist
CHECK: run `SHOW VARIABLES` / cluster state to confirm is_paused = false
before calling inject_database_initial_barrier.

match inject_database_initial_barrier(...).await {
    Err(e) if e.to_string().contains("should not pause when having snapshot backfill job") => {
        // resume database, then retry
        resume_database(db_id).await?;
        inject_database_initial_barrier(...).await?;
    }
    other => other?,
}

When it happens

Trigger: Injecting the database initial barrier while `is_paused` is true for a database containing snapshot backfill job {job_id} (and after confirming that job has no CDC backfill splits).

Common situations: Operator paused the cluster/database (e.g. via ALTER DATABASE ... PAUSE or during maintenance) and then triggered a snapshot backfill / database recovery; automation that pauses during upgrades while backfill jobs are in flight.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• batch refresh job {} has no snapshot backfill info
• database of cross db upstream table {} not found
• cannot reschedule jobs {:?} when creating jobs with unresche
• cannot create streaming job with snapshot backfill when paus
• missing lagging barriers for direct log-store start from sna

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/f16c5768bcd8aade. Report an issue: GitHub.