What causes "failed to wait for sink rebuild to finish" in risingwavelabs/risingwave?

Inspect logs for a panic in the sink reader between receiving the request and replying

failed to wait for sink rebuild to finish

Error message

failed to wait for sink rebuild to finish

What it means

After successfully sending a RebuildSink request, the executor awaits the oneshot reply from the reader confirming the sink was rebuilt with the new vnode bitmap. If the reply channel is dropped before a response is sent — the reader died mid-rebuild or was cancelled — the await yields RecvError and the executor surfaces this error.

Solutions

Inspect logs for a panic in the sink reader between receiving the request and replying
Trigger actor recovery (restart the sink actor); state should rebuild cleanly
Check for concurrent recovery/migration racing with the rebuild and serialize rebuild handling
If consistent, debug the rebuild path in sink.rs where the Responder is consumed and file an issue

Defensive patterns

Strategy: retry

Try / catch

match rx.await {
    Ok(responder_result) => responder_result,
    Err(_recv_err) => Err(anyhow!("failed to wait for sink rebuild to finish")),
}

Prevention

Wrap rebuild work in catch_unwind / ensure Responder is always replied, even on error
Avoid cancelling the reader mid-rebuild during recovery migrations
Test vnode-scaling barriers against sinks in CI

When it happens

Trigger: rx.await returns Err because the reader task dropped the Responder without replying: reader panicked inside rebuild, task cancelled during shutdown/recovery, or rebuild future was aborted.

Common situations: Panics inside sink rebuild logic, concurrent recovery cancelling the reader while a barrier-triggered rebuild is in flight, kill -9 / failover during a vnode scaling operation.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/81e84588861b96a4. Report an issue: GitHub.

Appendix: source

Thrown at src/stream/src/executor/sink.rs:535

                                schema_change,
                                wait_log_store_flush,
                            },
                        )
                        .await?;

                    let mutation = barrier.mutation.clone();
                    yield Message::Barrier(barrier);
                    if F::REBUILD_SINK_ON_UPDATE_VNODE_BITMAP
                        && let Some(new_vnode_bitmap) = update_vnode_bitmap.clone()
                    {
                        let (tx, rx) = oneshot::channel();
                        rebuild_sink_tx
                            .send(RebuildSinkMessage::RebuildSink(new_vnode_bitmap, tx))
                            .map_err(|_| {
                                anyhow!("failed to send the rebuild-sink request to the reader")
                            })?;
                        rx.await
                            .map_err(|_| anyhow!("failed to wait for sink rebuild to finish"))?;
                    }
                    post_flush.post_yield_barrier().await?;

                    if let Some(mutation) = mutation.as_deref() {
                        match mutation {
                            Mutation::Pause => {
                                log_writer.pause()?;
                                is_paused = true;
                            }
                            Mutation::Resume => {
                                log_writer.resume()?;
                                is_paused = false;
                            }
                            Mutation::Throttle(fragment_to_apply) => {
                                if let Some(entry) = fragment_to_apply.get(&fragment_id)
                                    && entry.throttle_type() == ThrottleType::Sink
                                {
                                    tracing::info!(

View on GitHub (pinned to 6469eb736d)