{"record":{"id":"81e84588861b96a4","repo":"risingwavelabs/risingwave","slug":"failed-to-wait-for-sink-rebuild-to-finish","errorCode":null,"errorMessage":"failed to wait for sink rebuild to finish","messagePattern":"failed to wait for sink rebuild to finish","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/stream/src/executor/sink.rs","lineNumber":535,"sourceCode":"                                schema_change,\n                                wait_log_store_flush,\n                            },\n                        )\n                        .await?;\n\n                    let mutation = barrier.mutation.clone();\n                    yield Message::Barrier(barrier);\n                    if F::REBUILD_SINK_ON_UPDATE_VNODE_BITMAP\n                        && let Some(new_vnode_bitmap) = update_vnode_bitmap.clone()\n                    {\n                        let (tx, rx) = oneshot::channel();\n                        rebuild_sink_tx\n                            .send(RebuildSinkMessage::RebuildSink(new_vnode_bitmap, tx))\n                            .map_err(|_| {\n                                anyhow!(\"failed to send the rebuild-sink request to the reader\")\n                            })?;\n                        rx.await\n                            .map_err(|_| anyhow!(\"failed to wait for sink rebuild to finish\"))?;\n                    }\n                    post_flush.post_yield_barrier().await?;\n\n                    if let Some(mutation) = mutation.as_deref() {\n                        match mutation {\n                            Mutation::Pause => {\n                                log_writer.pause()?;\n                                is_paused = true;\n                            }\n                            Mutation::Resume => {\n                                log_writer.resume()?;\n                                is_paused = false;\n                            }\n                            Mutation::Throttle(fragment_to_apply) => {\n                                if let Some(entry) = fragment_to_apply.get(&fragment_id)\n                                    && entry.throttle_type() == ThrottleType::Sink\n                                {\n                                    tracing::info!(","sourceCodeStart":517,"sourceCodeEnd":553,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/stream/src/executor/sink.rs#L517-L553","documentation":"After successfully sending a RebuildSink request, the executor awaits the oneshot reply from the reader confirming the sink was rebuilt with the new vnode bitmap. If the reply channel is dropped before a response is sent — the reader died mid-rebuild or was cancelled — the await yields RecvError and the executor surfaces this error.","triggerScenarios":"rx.await returns Err because the reader task dropped the Responder without replying: reader panicked inside rebuild, task cancelled during shutdown/recovery, or rebuild future was aborted.","commonSituations":"Panics inside sink rebuild logic, concurrent recovery cancelling the reader while a barrier-triggered rebuild is in flight, kill -9 / failover during a vnode scaling operation.","solutions":["Inspect logs for a panic in the sink reader between receiving the request and replying","Trigger actor recovery (restart the sink actor); state should rebuild cleanly","Check for concurrent recovery/migration racing with the rebuild and serialize rebuild handling","If consistent, debug the rebuild path in sink.rs where the Responder is consumed and file an issue"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match rx.await {\n    Ok(responder_result) => responder_result,\n    Err(_recv_err) => Err(anyhow!(\"failed to wait for sink rebuild to finish\")),\n}","preventionTips":["Wrap rebuild work in catch_unwind / ensure Responder is always replied, even on error","Avoid cancelling the reader mid-rebuild during recovery migrations","Test vnode-scaling barriers against sinks in CI"],"tags":["stream-executor","sink","oneshot-channel"],"backgroundTag":"broken-pipe","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}