risingwavelabs/risingwave · error

Unexpected backfill state

Error message

Unexpected backfill state: {:?}

What it means

During a split-change (split migration) handled in the forwarding stage, the source backfill executor expects every migrated split's backfill state to be Finished. Any other BackfillState means an inconsistent internal state machine, so update_state_if_changed_forward_stage fails with this error.

Solutions

1. Capture the full backfill state from the error message and file an issue — this indicates an internal invariant violation.
2. Retry the MV after recovery; if it persists, recreate the materialized view to reset backfill state.
3. Check recent parallelism/split change operations on the source and whether they occurred during backfill.

// before split changes, confirm all splits report backfill finished:
// query backfill progress/state from system catalogs or metrics before rescaling

fn is_finished(state: &BackfillState) -> bool {
    matches!(state.state, BackfillState::Finished)
}

match res.backfill_state {
    Some(s) if matches!(s.state, BackfillState::Finished) => proceed(),
    Some(s) => report_invariant_violation(format!("Unexpected backfill state: {:?}", s)),
    None => proceed(),
}

When it happens

Trigger: apply_split_change_forward_stage_after_yield_barrier encounters a split whose backfill_state.state is not BackfillState::Finished while processing the forwarding stage after a split change.

Common situations: Split change raced with backfill completion, state corrupted or partially persisted across recovery, or a bug/edge case in the backfill state machine with parallelism changes.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• ALTER TABLE SET BACKFILL PARALLELISM is not supported for…
• Backfill order strategy has a cycle
• Backfill progress for vnode
• batch refresh job has no snapshot backfill info
• Cannot alter the job

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/f223316b6872918a. Report an issue: GitHub.

Appendix: source

Thrown at src/stream/src/executor/source/source_backfill_executor.rs:1167

        // Checks added splits
        for split_id in &target_splits {
            if !states.contains_key(split_id) {
                split_changed = true;

                let backfill_state = committed_reader
                    .try_recover_from_state_store(split_id)
                    .await?;
                match &backfill_state {
                    None => {
                        // Newly added split. We don't need to backfill!
                        newly_added_splits.push(split_id.clone());
                    }
                    Some(backfill_state) => {
                        // Migrated split. It should also be finished since we are in forwarding stage.
                        match backfill_state.state {
                            BackfillState::Finished => {}
                            _ => {
                                return Err(anyhow::anyhow!(
                                    "Unexpected backfill state: {:?}",
                                    backfill_state
                                )
                                .into());
                            }
                        }
                    }
                }
                states.insert(
                    split_id.clone(),
                    backfill_state.unwrap_or(BackfillStateWithProgress {
                        state: BackfillState::Finished,
                        num_consumed_rows: 0,
                        target_offset: None,
                    }),
                );
            }
        }

View on GitHub (pinned to 6469eb736d)