risingwavelabs/risingwave · error

cannot reschedule jobs {:?} when creating jobs with unresche

Error message

cannot reschedule jobs {:?} when creating jobs with unreschedulable backfill fragments

What it means

When creating jobs contain snapshot-backfill fragments that cannot be rescheduled, any reschedule command touching those blocked job IDs is rejected at barrier time and reported as a failed start. This protects consistency of snapshot backfill, which cannot survive actor migration mid-creation.

Source

Thrown at src/meta/src/barrier/checkpoint/control.rs:1278

            ..
        }) = &command
            && !self.independent_checkpoint_job_controls.is_empty()
        {
            let blocked_job_ids =
                self.collect_reschedule_blocked_jobs_for_independent_jobs_inflight()?;
            let blocked_reschedule_job_ids = self.collect_reschedule_blocked_job_ids(
                &reschedule_plan.reschedules,
                &reschedule_plan.fragment_actors,
                &blocked_job_ids,
            );
            if !blocked_reschedule_job_ids.is_empty() {
                warn!(
                    blocked_reschedule_job_ids = ?blocked_reschedule_job_ids,
                    "reject reschedule fragments related to creating unreschedulable backfill jobs"
                );
                if let Some(notifier) = notifier_start {
                    notifier.notify_start_failed(
                        anyhow!(
                            "cannot reschedule jobs {:?} when creating jobs with unreschedulable backfill fragments",
                            blocked_reschedule_job_ids
                        )
                            .into(),
                    );
                }
                return Ok(());
            }
        }

        if !matches!(&command, Some(Command::CreateStreamingJob { .. }))
            && self.database_info.is_empty()
        {
            assert!(
                self.independent_checkpoint_job_controls.is_empty(),
                "should not have snapshot backfill job when there is no normal job in database"
            );
            // Drop the guard to remove the metric series of this database.

View on GitHub (pinned to 6469eb736d)

Solutions

1. Wait until the snapshot-backfill job finishes creating, then retry the reschedule.
2. Exclude the blocked job IDs (printed in the message) from the reschedule plan and reschedule only unrelated jobs.
3. Cancel the creating backfill job if the reschedule is urgent, reschedule, then recreate the job.

// before rescheduling, ensure no creating snapshot-backfill jobs intersect the plan
let creating_backfill: HashSet<_> = get_creating_jobs_with_unreschedulable_backfill().await?;
assert!(plan.job_ids.iter().all(|j| !creating_backfill.contains(j)), "plan touches blocked jobs");

When it happens

Trigger: Issuing a reschedule (scale/adjust parallelism) while there are creating streaming jobs with unreschedulable snapshot backfill fragments, where the reschedule plan includes the blocked job IDs during handle_new_barrier.

Common situations: Running ALTER ... parallelism / auto-scaling while a materialized view with snapshot backfill is still being created; mixed workloads of DDL and scaling operations.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• duplicate worker id {worker_id} in plan, prev {worker_id} ->
• database of cross db upstream table {} not found
• cannot create streaming job with snapshot backfill when paus
• missing lagging barriers for direct log-store start from sna
• reschedule intent must be resolved before apply

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/27c4295a92963915. Report an issue: GitHub.