risingwavelabs/risingwave · error · StreamExecutorError

Backfill progress for vnode

Error message

Backfill progress for vnode {:#?} not found, backfill_state not initialized properly

What it means

A lookup of backfill progress for a specific vnode found no entry in the `BackfillProgress` map. Every vnode the executor is responsible for must be initialized with progress state before chunks are read or marked; a missing entry means the state was never initialized from the progress table.

Solutions

1. Verify the progress table contains one row per vnode for this actor's vnode range and reinitialize missing entries from defaults.
2. Check whether vnode mapping (parallelism) changed since the progress table was written; migrate or recreate the backfill state.
3. Ensure the state is initialized via the init path before any snapshot read; if the bug is in initialization order, file an issue with the trace.

// before reading/chunk-marking
for vnode in assigned_vnodes { debug_assert!(progress.contains(vnode), "missing progress for {vnode:?}"); }

fn get_progress_safe<'a>(p: &'a BackfillProgress, vnode: &VirtualNode) -> Option<&'a BackfillProgressPerVnode> { p.inner.get(vnode).map(|e| e.current_state()) }

let progress = state.get_progress(&vnode).with_context(|| format!("vnode {vnode:?} missing; reinit state"))?;

When it happens

Trigger: `get_progress` is called (from `snapshot_read_per_vnode` or `mark_chunk_ref_by_vnode`) for a vnode absent from `self.inner`, e.g. state built from progress-table rows that don't cover all vnodes of the fragment, or vnode mapping changed after recovery.

Common situations: Progress table missing rows for some vnodes (partial write/corruption); changed parallelism/vnode mapping across restarts; code bug initializing state before first chunk arrives.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• failed to find fragment
• invalid backfill state: backfill_finished
• invalid backfill state: row_count
• invalid backfill state: unfinished row has null cdc_offset
• mismatch initial vnode bitmap

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/c36bebe8cdedec02. Report an issue: GitHub.

Appendix: source

Thrown at src/stream/src/executor/backfill/utils.rs:80

            )
        })
    }

    pub(crate) fn get_current_state(
        &mut self,
        vnode: &VirtualNode,
    ) -> &mut BackfillProgressPerVnode {
        &mut self.inner.get_mut(vnode).unwrap().current_state
    }

    // Expects the vnode to always have progress, otherwise it will return an error.
    pub(crate) fn get_progress(
        &self,
        vnode: &VirtualNode,
    ) -> StreamExecutorResult<&BackfillProgressPerVnode> {
        match self.inner.get(vnode) {
            Some(p) => Ok(p.current_state()),
            None => bail!(
                "Backfill progress for vnode {:#?} not found, backfill_state not initialized properly",
                vnode,
            ),
        }
    }

    pub(crate) fn update_progress(
        &mut self,
        vnode: VirtualNode,
        new_pos: OwnedRow,
        snapshot_row_count_delta: u64,
    ) -> StreamExecutorResult<()> {
        let state = self.get_current_state(&vnode);
        match state {
            BackfillProgressPerVnode::NotStarted => {
                *state = BackfillProgressPerVnode::InProgress {
                    current_pos: new_pos,
                    snapshot_row_count: snapshot_row_count_delta,

View on GitHub (pinned to 6469eb736d)