risingwavelabs/risingwave · error · anyhow::Error

newly start epoch {} after update vnode bitmap not matched w

Error message

newly start epoch {} after update vnode bitmap not matched with prev_epoch {}

What it means

After applying a new vnode bitmap via coordinator_stream_handle.update_vnode_bitmap, the handle returns the epoch at which the update took effect. The coordinator requires this epoch to equal prev_epoch (the epoch the stream will resume from); a mismatch means the sink coordination state and the stream's epoch are out of sync after scaling.

Source

Thrown at src/connector/src/sink/coordinate.rs:244

                                    is_stop,
                                    "schema change should stop current sink for sink {}",
                                    self.param.sink_id
                                );
                            }
                            coordinator_stream_handle
                                .commit(epoch, metadata, schema_change)
                                .await?;
                            sink_writer_metrics
                                .sink_commit_duration
                                .observe(start_time.elapsed().as_secs_f64());

                            current_checkpoint = 0;
                            if let Some(new_vnode_bitmap) = new_vnode_bitmap {
                                let epoch = coordinator_stream_handle
                                    .update_vnode_bitmap(&new_vnode_bitmap)
                                    .await?;
                                if epoch != prev_epoch {
                                    bail!(
                                        "newly start epoch {} after update vnode bitmap not matched with prev_epoch {}",
                                        epoch,
                                        prev_epoch
                                    );
                                }
                            }
                            if is_stop {
                                coordinator_stream_handle.stop().await?;
                                info!(
                                    sink_id = %self.param.sink_id,
                                    "coordinated log sinker stops"
                                );
                                log_reader.truncate(TruncateOffset::Barrier { epoch })?;
                                return pending().await;
                            }
                            log_reader.truncate(TruncateOffset::Barrier { epoch })?;
                        } else {
                            let metadata = sink_writer.barrier(false).await?;

View on GitHub (pinned to 6469eb736d)

Solutions

1. Retry the recovery/scale operation; transient epoch mismatch often resolves once no scale is in flight.
2. Check meta logs to see why update_vnode_bitmap committed at a different epoch than prev_epoch.
3. Recover from an earlier consistent checkpoint so prev_epoch matches the bitmap-update epoch.

if err.to_string().contains("not matched with prev_epoch") {
    // wait for in-flight rescale to settle, then restart recovery from last checkpoint
}

When it happens

Trigger: During consume_log_and_sink's scale-in/out handling: new_vnode_bitmap is Some, update_vnode_bitmap(new_vnode_bitmap) returns an epoch that differs from prev_epoch.

Common situations: Sink rescaling (fragment parallelism change) concurrent with recovery; meta returned a commit epoch that differs from the epoch recorded before the bitmap update due to concurrent barrier/scale operations.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• should have meta client
• initial epoch {} greater than aligned initial epoch {}
• should get metadata on checkpoint barrier
• get none metadata in commit response for coordinated sink wr
• should get start response but get {:?}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d2795c103eb6f1ca. Report an issue: GitHub.