risingwavelabs/risingwave · error

legacy no-shuffle backfill recovered unfinished progress…

Error message

legacy no-shuffle backfill recovered unfinished progress; cancel and recreate the streaming job. upstream_table_id={:?}, fragment_id={:?}, actor_id={}, current_pos={:?}, row_count={}

What it means

In the legacy no-shuffle backfill executor, if recovery finds unfinished backfill progress, the executor refuses to run: legacy recovery cannot safely resume, so it bails and tells the operator to cancel and recreate the streaming job. This is a deliberate hard stop, not a transient failure.

Solutions

1. Cancel the streaming job and recreate it (as the message says) so backfill restarts cleanly
2. Migrate the job to the current snapshot-based backfill executor by recreating the MV/table on a current version
3. Check the release notes for the legacy no-shuffle backfill removal; plan recreation during upgrade
4. If recreating is too costly, contact support/restore from a backup taken before the crash

// Check whether the job uses the legacy no-shuffle backfill before relying on crash recovery
// (job created before the snapshot/shuffled backfill redesign => recovery of unfinished progress is unsupported)

match executor.execute().await {
    Err(e) if e.to_string().contains("legacy no-shuffle backfill") => {
        // cancel and recreate the streaming job
    }
    other => other?,
}

When it happens

Trigger: Starting/recovering a streaming job that uses the legacy no-shuffle backfill executor while its persisted state has is_finished=false (a crash or scale-down occurred mid-backfill).

Common situations: Old jobs created before the shuffle/snapshot backfill redesign that crashed during backfill and are recovered on cluster restart; recovering a legacy job after failover.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• corrupt MATCH_RECOGNIZE state row: seq column must be a…
• end of stream
• invalid backfill state: backfill_finished
• invalid backfill state: row_count
• invalid backfill state: unfinished row has null cdc_offset

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/5e668b500a6d4666. Report an issue: GitHub.

Appendix: source

Thrown at src/stream/src/executor/backfill/no_shuffle_backfill.rs:128

        let first_barrier = expect_first_barrier(&mut upstream).await?;
        let first_epoch = first_barrier.epoch;
        // The first barrier message should be propagated.
        yield Message::Barrier(first_barrier);

        if let Some(state_table) = self.state_table.as_mut() {
            state_table.init_epoch(first_epoch).await?;
        }

        let BackfillState {
            current_pos,
            is_finished,
            row_count,
            ..
        } = Self::recover_backfill_state(self.state_table.as_ref(), pk_indices.len()).await?;
        tracing::trace!(is_finished, row_count, "backfill state recovered");

        if !is_finished {
            bail!(
                "legacy no-shuffle backfill recovered unfinished progress; cancel and recreate the streaming job. upstream_table_id={:?}, fragment_id={:?}, actor_id={}, current_pos={:?}, row_count={}",
                upstream_table_id,
                self.fragment_id,
                self.actor_id,
                current_pos,
                row_count,
            );
        }

        tracing::trace!("Backfill has finished, waiting for barrier");

        // Wait for first barrier to come after backfill is finished.
        // So we can update our progress + persist the status.
        while let Some(Ok(msg)) = upstream.next().await {
            if let Some(msg) = mapping_message(msg, &self.output_indices) {
                if let Message::Barrier(barrier) = &msg {
                    // If already finished, no need persist any state, but we need to advance the
                    // epoch of the state table anyway.

View on GitHub (pinned to 6469eb736d)