risingwavelabs/risingwave · error

mismatch initial vnode bitmap: {:?}, expect: {:?}

Error message

mismatch initial vnode bitmap: {:?}, expect: {:?}

What it means

check_initial_vnode_bitmap verifies that the vnode bitmap announced by the upstream at the start of backfill matches what this backfill actor computed for itself. A mismatch means distribution changed between planning and execution, so parallel backfill would read/assign wrong key ranges.

Source

Thrown at src/stream/src/executor/backfill/snapshot_backfill/consume_upstream/upstream_table_trait.rs:121

        vnode: VirtualNode,
        epoch: u64,
        start_pk: Option<OwnedRow>,
    ) -> StreamExecutorResult<Self::ChangeLogStream> {
        let stream = self
            .batch_iter_vnode_log(
                epoch,
                HummockReadEpoch::Committed(epoch),
                start_pk.as_ref(),
                vnode,
            )
            .await?;
        Ok(stream.map_err(Into::into))
    }

    fn check_initial_vnode_bitmap(&self, vnodes: &Bitmap) -> StreamExecutorResult<()> {
        let expected_vnodes = &**self.vnodes();
        if expected_vnodes != vnodes {
            Err(anyhow!(
                "mismatch initial vnode bitmap: {:?}, expect: {:?}",
                vnodes,
                self.vnodes()
            )
            .into())
        } else {
            Ok(())
        }
    }

    fn update_vnode_bitmap(&mut self, new_vnodes: Arc<Bitmap>) {
        let _ = self.update_vnode_bitmap(new_vnodes);
    }
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Retry the job recovery/rescheduling so the actor is rebuilt with the current upstream distribution
2. Cancel and recreate the MV/backfill job so planning and execution see the same distribution
3. Avoid scaling the cluster while backfill jobs are being created; rescale after backfill completes
4. If reproducible, file a bug with the fragment and actor IDs from the error

if let Err(e) = executor.check_initial_vnode_bitmap(&vnodes) {
    // treat as transient scheduling mismatch: recreate/recover the actor
    // if persistent, cancel and recreate the MV
}

When it happens

Trigger: The upstream table's vnode bitmap differs from the backfill executor's own vnodes at actor startup, typically after a scale-in/out or fragment rescheduling between job creation and actor start, or a bug in fragment distribution of a parallelized job.

Common situations: Rescaling a cluster while a backfill job is being created/recovered; inconsistent parallelism settings between upstream and backfill fragments; meta bug in fragment mapping after failover.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• should not update vnode bitmap during consuming log store
• legacy no-shuffle backfill recovered unfinished progress; ca
• end of stream
• no upstream while snapshot epoch not set
• locality provider upstream ended unexpectedly during backfil

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d35cdfb22a5e48fe. Report an issue: GitHub.