{"record":{"id":"5e668b500a6d4666","repo":"risingwavelabs/risingwave","slug":"legacy-no-shuffle-backfill-recovered-unfinished-pr","errorCode":null,"errorMessage":"legacy no-shuffle backfill recovered unfinished progress; cancel and recreate the streaming job. upstream_table_id={:?}, fragment_id={:?}, actor_id={}, current_pos={:?}, row_count={}","messagePattern":"legacy no-shuffle backfill recovered unfinished progress; cancel and recreate the streaming job\\. upstream_table_id=(.+?), fragment_id=(.+?), actor_id=(.+?), current_pos=(.+?), row_count=(.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/stream/src/executor/backfill/no_shuffle_backfill.rs","lineNumber":128,"sourceCode":"        let first_barrier = expect_first_barrier(&mut upstream).await?;\n        let first_epoch = first_barrier.epoch;\n        // The first barrier message should be propagated.\n        yield Message::Barrier(first_barrier);\n\n        if let Some(state_table) = self.state_table.as_mut() {\n            state_table.init_epoch(first_epoch).await?;\n        }\n\n        let BackfillState {\n            current_pos,\n            is_finished,\n            row_count,\n            ..\n        } = Self::recover_backfill_state(self.state_table.as_ref(), pk_indices.len()).await?;\n        tracing::trace!(is_finished, row_count, \"backfill state recovered\");\n\n        if !is_finished {\n            bail!(\n                \"legacy no-shuffle backfill recovered unfinished progress; cancel and recreate the streaming job. upstream_table_id={:?}, fragment_id={:?}, actor_id={}, current_pos={:?}, row_count={}\",\n                upstream_table_id,\n                self.fragment_id,\n                self.actor_id,\n                current_pos,\n                row_count,\n            );\n        }\n\n        tracing::trace!(\"Backfill has finished, waiting for barrier\");\n\n        // Wait for first barrier to come after backfill is finished.\n        // So we can update our progress + persist the status.\n        while let Some(Ok(msg)) = upstream.next().await {\n            if let Some(msg) = mapping_message(msg, &self.output_indices) {\n                if let Message::Barrier(barrier) = &msg {\n                    // If already finished, no need persist any state, but we need to advance the\n                    // epoch of the state table anyway.","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/stream/src/executor/backfill/no_shuffle_backfill.rs#L110-L146","documentation":"In the legacy no-shuffle backfill executor, if recovery finds unfinished backfill progress, the executor refuses to run: legacy recovery cannot safely resume, so it bails and tells the operator to cancel and recreate the streaming job. This is a deliberate hard stop, not a transient failure.","triggerScenarios":"Starting/recovering a streaming job that uses the legacy no-shuffle backfill executor while its persisted state has is_finished=false (a crash or scale-down occurred mid-backfill).","commonSituations":"Old jobs created before the shuffle/snapshot backfill redesign that crashed during backfill and are recovered on cluster restart; recovering a legacy job after failover.","solutions":["Cancel the streaming job and recreate it (as the message says) so backfill restarts cleanly","Migrate the job to the current snapshot-based backfill executor by recreating the MV/table on a current version","Check the release notes for the legacy no-shuffle backfill removal; plan recreation during upgrade","If recreating is too costly, contact support/restore from a backup taken before the crash"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Check whether the job uses the legacy no-shuffle backfill before relying on crash recovery\n// (job created before the snapshot/shuffled backfill redesign => recovery of unfinished progress is unsupported)","typeGuard":null,"tryCatchPattern":"match executor.execute().await {\n    Err(e) if e.to_string().contains(\"legacy no-shuffle backfill\") => {\n        // cancel and recreate the streaming job\n    }\n    other => other?,\n}","preventionTips":["Recreate legacy backfill jobs on the modern executor before production use","Monitor backfill completion so jobs are not left unfinished across restarts"],"tags":["rust","streaming","backfill","legacy","recovery"],"backgroundTag":"invalid-state-transition","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}