What causes "failed to receive update vnode" in risingwavelabs/risingwave?

Check why the vnode-bitmap sender side terminated before sending (upstream actor logs).

failed to receive update vnode

Error message

failed to receive update vnode

What it means

This is the other branch of KvLogStoreReader::init: when the reader has no init-epoch oneshot, it instead expects the first message on `update_vnode_bitmap_rx` to carry a new vnode bitmap and write epoch. Receiving None means all senders of that channel were dropped, so the reader never learned its vnode mapping and cannot initialize its deserializer/epoch baseline.

Source

Thrown at src/stream/src/common/log_store_impl/kv_log_store/reader.rs:341

    async fn init(&mut self) -> LogStoreResult<()> {
        if let Some(init_epoch_rx) = self.init_epoch_rx.take() {
            let init_epoch = init_epoch_rx
                .await
                .map_err(|_| anyhow!("should get the first epoch"))?;
            let first_write_epoch = init_epoch.curr;

            assert_eq!(
                self.first_write_epoch.replace(first_write_epoch),
                None,
                "should not init twice"
            );
        } else {
            let (new_vnode_bitmap, write_epoch) = self
                .update_vnode_bitmap_rx
                .recv()
                .await
                .ok_or_else(|| anyhow!("failed to receive update vnode"))?;
            self.state.serde.update_vnode_bitmap(new_vnode_bitmap);
            self.first_write_epoch = Some(write_epoch);
        };

        self.future_state = KvLogStoreReaderFutureState::Reset;
        self.latest_offset = None;
        self.truncate_offset = None;
        self.rewind_delay = RewindDelay::new(&self.metrics);

        Ok(())
    }

    async fn next_item(&mut self) -> LogStoreResult<(u64, LogStoreReadItem)> {
        while *self.is_paused.borrow_and_update() {
            info!("next_item of {} get blocked by is_pause", self.identity);
            self.is_paused
                .changed()
                .instrument_await("Wait for Pause Resume")

View on GitHub (pinned to 6469eb736d)

Solutions

Check why the vnode-bitmap sender side terminated before sending (upstream actor logs).
Ensure reader init happens only while the vnode-update sender is alive and running.
Verify the correct init mode: either a writer oneshot or a live vnode-bitmap channel must exist.
Treat as terminal for the reader and rebuild it through recovery.

Example fix

// before: reader initialized after upstream dropped the vnode channel
drop(vnode_update_tx);
reader.init().await?;

// after: keep sender alive until reader init completes
reader.init().await?;
drop(vnode_update_tx);

Defensive patterns

Strategy: try-catch

Validate before calling

if vnode_update_tx.is_closed() {
    fail_fast("vnode update sender gone before reader init");
}

Try / catch

if let Err(e) = reader.init().await {
    if e.to_string().contains("failed to receive update vnode") {
        // upstream dropped the bitmap channel; rebuild reader via recovery
        rebuild_reader();
    }
    return Err(e);
}

Prevention

Keep the vnode-bitmap sender alive until reader init completes.
Initialize the reader while its upstream executor is still running.
Pick exactly one init mode (writer oneshot or vnode channel) consistently.

When it happens

Trigger: Reader init without an init_epoch_rx, but the component that sends the vnode bitmap update (the upstream executor / merge owner) dropped its sender before delivering the bitmap — e.g. it terminated or was reconfigured away.

Common situations: Upstream executor failure during reader initialization; scaling/migration paths where the vnode update sender is replaced but the old reader persists; ordering bugs where reader init runs after upstream teardown.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/0aca3da9334a5410. Report an issue: GitHub.