risingwavelabs/risingwave · error

legacy no-shuffle backfill recovered unfinished progress; ca

Error message

legacy no-shuffle backfill recovered unfinished progress; cancel and recreate the streaming job. upstream_table_id={:?}, fragment_id={:?}, actor_id={}, current_pos={:?}, row_count={}

What it means

In the legacy no-shuffle backfill executor, if recovery finds unfinished backfill progress, the executor refuses to run: legacy recovery cannot safely resume, so it bails and tells the operator to cancel and recreate the streaming job. This is a deliberate hard stop, not a transient failure.

Source

Thrown at src/stream/src/executor/backfill/no_shuffle_backfill.rs:128

        let first_barrier = expect_first_barrier(&mut upstream).await?;
        let first_epoch = first_barrier.epoch;
        // The first barrier message should be propagated.
        yield Message::Barrier(first_barrier);

        if let Some(state_table) = self.state_table.as_mut() {
            state_table.init_epoch(first_epoch).await?;
        }

        let BackfillState {
            current_pos,
            is_finished,
            row_count,
            ..
        } = Self::recover_backfill_state(self.state_table.as_ref(), pk_indices.len()).await?;
        tracing::trace!(is_finished, row_count, "backfill state recovered");

        if !is_finished {
            bail!(
                "legacy no-shuffle backfill recovered unfinished progress; cancel and recreate the streaming job. upstream_table_id={:?}, fragment_id={:?}, actor_id={}, current_pos={:?}, row_count={}",
                upstream_table_id,
                self.fragment_id,
                self.actor_id,
                current_pos,
                row_count,
            );
        }

        tracing::trace!("Backfill has finished, waiting for barrier");

        // Wait for first barrier to come after backfill is finished.
        // So we can update our progress + persist the status.
        while let Some(Ok(msg)) = upstream.next().await {
            if let Some(msg) = mapping_message(msg, &self.output_indices) {
                if let Message::Barrier(barrier) = &msg {
                    // If already finished, no need persist any state, but we need to advance the
                    // epoch of the state table anyway.

View on GitHub (pinned to 6469eb736d)

Solutions

1. Cancel the streaming job and recreate it (as the message says) so backfill restarts cleanly
2. Migrate the job to the current snapshot-based backfill executor by recreating the MV/table on a current version
3. Check the release notes for the legacy no-shuffle backfill removal; plan recreation during upgrade
4. If recreating is too costly, contact support/restore from a backup taken before the crash

// Check whether the job uses the legacy no-shuffle backfill before relying on crash recovery
// (job created before the snapshot/shuffled backfill redesign => recovery of unfinished progress is unsupported)

match executor.execute().await {
    Err(e) if e.to_string().contains("legacy no-shuffle backfill") => {
        // cancel and recreate the streaming job
    }
    other => other?,
}

When it happens

Trigger: Starting/recovering a streaming job that uses the legacy no-shuffle backfill executor while its persisted state has is_finished=false (a crash or scale-down occurred mid-backfill).

Common situations: Old jobs created before the shuffle/snapshot backfill redesign that crashed during backfill and are recovered on cluster restart; recovering a legacy job after failover.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• invalid backfill state: row_count
• invalid backfill state: backfill_finished
• invalid backfill state: unfinished row has null cdc_offset
• end of stream
• mismatch initial vnode bitmap: {:?}, expect: {:?}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/5e668b500a6d4666. Report an issue: GitHub.