risingwavelabs/risingwave · error

should not update vnode bitmap during consuming log store

Error message

should not update vnode bitmap during consuming log store

What it means

While the backfill executor is in the log-store-consuming phase (after snapshot backfill finishes), it commits each barrier. An update-vnode-bitbarrier in this phase would invalidate the ongoing consumption, which the design does not support, so the executor returns this invariant error.

Source

Thrown at src/stream/src/executor/backfill/snapshot_backfill/executor.rs:428

                                if is_finished {
                                    assert_eq!(upstream_buffer.pending_epoch_lag(), 0);
                                    assert!(pending_non_checkpoint_barrier.is_empty());
                                    self.progress.finish_consuming_log_store(barrier.epoch);
                                } else {
                                    self.progress.update_create_mview_log_store_progress(
                                        barrier.epoch,
                                        upstream_buffer.pending_epoch_lag(),
                                    );
                                }
                            }

                            let post_commit = backfill_state.commit(barrier.epoch).await?;
                            let update_vnode_bitmap =
                                barrier.as_update_vnode_bitmap(self.actor_ctx.id);
                            yield Message::Barrier(barrier);
                            post_commit.post_yield_barrier(None).await?;
                            if update_vnode_bitmap.is_some() {
                                return Err(anyhow!(
                                    "should not update vnode bitmap during consuming log store"
                                )
                                .into());
                            }

                            if is_finished {
                                assert!(
                                    pending_non_checkpoint_barrier.is_empty(),
                                    "{pending_non_checkpoint_barrier:?}"
                                );
                                break;
                            }
                        }
                        trace!(
                            ?barrier_epoch,
                            table_id = %self.upstream_table.table_id(),
                            "finish consuming log store"
                        );

View on GitHub (pinned to 6469eb736d)

Solutions

1. Wait until backfill completes before scaling the cluster; retry the job after rescheduling with a stable topology
2. Cancel and recreate the streaming job so backfill runs under the new vnode assignment from the start
3. Check meta logs for the scale-in/out command that triggered the vnode update and reschedule it after backfill
4. If frequent, upgrade to a version supporting rescheduling during backfill (shuffled backfill)

// Guard at the operational level: defer scale-in/out while backfill is in the log-store phase
if backfill_in_progress && pending_scale_command { postpone_scale(); }

if let Err(e) = run_backfill().await {
    if e.to_string().contains("update vnode bitmap during consuming log store") {
        // recreate the job so backfill runs under the new vnode layout
    }
}

When it happens

Trigger: A barrier carrying an update_vnode_bitmap (from a cluster scale-in/out or upstream parallelism change) arrives while this actor is consuming the log store phase of snapshot backfill.

Common situations: Rescaling the cluster (adding/removing nodes) during the log-store phase of a snapshot backfill; upstream parallelism change mid-backfill.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• mismatch initial vnode bitmap: {:?}, expect: {:?}
• downstream relation missing for {} -> {}
• legacy no-shuffle backfill recovered unfinished progress; ca
• end of stream
• no upstream while snapshot epoch not set

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/a1174e815cc32dac. Report an issue: GitHub.