{"record":{"id":"b1266b28bc0c31ca","repo":"risingwavelabs/risingwave","slug":"failed-to-send-request-to","errorCode":null,"errorMessage":"failed to send request to {} {:?}","messagePattern":"failed to send request to (.+?) (.+?)","errorType":"error_code","errorClass":"MetaError","httpStatus":null,"severity":"error","filePath":"src/meta/src/barrier/rpc.rs","lineNumber":1370,"sourceCode":"                                                                    .collect(),\n                                                                dispatchers,\n                                                                vnode_bitmap: actor.vnode_bitmap.map(|bitmap| bitmap.to_protobuf()),\n                                                                mview_definition: actor.mview_definition,\n                                                                expr_context: actor.expr_context,\n                                                                config_override: actor.config_override.to_string(),\n                                                                initial_subscriber_ids: initial_subscriber_ids.iter().copied().collect(),\n                                                            }\n                                                        })\n                                                        .collect(),\n                                                }\n                                            })\n                                            .collect(),\n                                    },\n                                ),\n                            ),\n                        })\n                        .map_err(|_| {\n                            MetaError::from(anyhow!(\n                                \"failed to send request to {} {:?}\",\n                                node.worker_id,\n                                node.host\n                            ))\n                        })?;\n\n                    node_need_collect.insert(*worker_id);\n                    Result::<_, MetaError>::Ok(())\n                }\n            })\n            .inspect_err(|e| {\n                // Record failure in event log.\n                use risingwave_pb::meta::event_log;\n                let event = event_log::EventInjectBarrierFail {\n                    prev_epoch: barrier_info.prev_epoch(),\n                    cur_epoch: barrier_info.curr_epoch(),\n                    error: e.to_report_string(),\n                };","sourceCodeStart":1352,"sourceCodeEnd":1388,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/rpc.rs#L1352-L1388","documentation":"`inject_barrier` sends the barrier/mutation request over the worker's gRPC control stream; the `send` future returns `Err` when the stream is closed or broken. The code maps that to `MetaError::from(anyhow!(\"failed to send request to {} {:?}\", node.worker_id, node.host))`, including the worker id and host for diagnosis. This signals the control stream to that worker broke while sending a barrier.","triggerScenarios":"`inject_barrier` calling `control_stream.send(...)` (via the request builder) and the underlying channel returns an error — stream closed by the worker, connection reset, or worker process exit mid-send.","commonSituations":"Worker crash or OOM-kill during barrier injection; network interruption between meta and compute node; long-running barrier send across a node restart; madsim/fault-injection tests simulating stream failure.","solutions":["Restart/reconnect the affected worker node; the barrier manager's retry/recovery will re-inject the barrier.","Check worker logs at node.worker_id / node.host for the crash or stream-close reason (OOM, panic, deploy restart).","Inspect network stability (timeouts, LB idle disconnects) between meta and worker hosts.","If persistent, verify worker and meta versions are compatible and the control stream protocol matches."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Pre-flight: confirm the stream is open before sending\n// (control streams expose closed state; check before each send)\nif control_stream.is_closed() {\n    // mark worker disconnected and skip/retry\n}","typeGuard":null,"tryCatchPattern":"match inject_barrier(mutation).await {\n    Err(e) if e.to_string().contains(\"failed to send request to\") => {\n        tracing::warn!(\"control stream broken, awaiting recovery: {e}\");\n        // barrier manager will retry after worker reconnects\n    }\n    other => other?,\n}","preventionTips":["Enable keepalives and sane idle timeouts on gRPC control streams.","Monitor worker restarts and OOM kills; they manifest as this error.","Ensure barrier send failures trigger the standard recovery/retry path rather than manual intervention."],"tags":["grpc","stream","barrier","network"],"backgroundTag":"broken-pipe","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}