risingwavelabs/risingwave · error · MetaError

failed to send request to {} {:?}

Error message

failed to send request to {} {:?}

What it means

`inject_barrier` sends the barrier/mutation request over the worker's gRPC control stream; the `send` future returns `Err` when the stream is closed or broken. The code maps that to `MetaError::from(anyhow!("failed to send request to {} {:?}", node.worker_id, node.host))`, including the worker id and host for diagnosis. This signals the control stream to that worker broke while sending a barrier.

Source

Thrown at src/meta/src/barrier/rpc.rs:1370

                                                                    .collect(),
                                                                dispatchers,
                                                                vnode_bitmap: actor.vnode_bitmap.map(|bitmap| bitmap.to_protobuf()),
                                                                mview_definition: actor.mview_definition,
                                                                expr_context: actor.expr_context,
                                                                config_override: actor.config_override.to_string(),
                                                                initial_subscriber_ids: initial_subscriber_ids.iter().copied().collect(),
                                                            }
                                                        })
                                                        .collect(),
                                                }
                                            })
                                            .collect(),
                                    },
                                ),
                            ),
                        })
                        .map_err(|_| {
                            MetaError::from(anyhow!(
                                "failed to send request to {} {:?}",
                                node.worker_id,
                                node.host
                            ))
                        })?;

                    node_need_collect.insert(*worker_id);
                    Result::<_, MetaError>::Ok(())
                }
            })
            .inspect_err(|e| {
                // Record failure in event log.
                use risingwave_pb::meta::event_log;
                let event = event_log::EventInjectBarrierFail {
                    prev_epoch: barrier_info.prev_epoch(),
                    cur_epoch: barrier_info.curr_epoch(),
                    error: e.to_report_string(),
                };

View on GitHub (pinned to 6469eb736d)

Solutions

1. Restart/reconnect the affected worker node; the barrier manager's retry/recovery will re-inject the barrier.
2. Check worker logs at node.worker_id / node.host for the crash or stream-close reason (OOM, panic, deploy restart).
3. Inspect network stability (timeouts, LB idle disconnects) between meta and worker hosts.
4. If persistent, verify worker and meta versions are compatible and the control stream protocol matches.

// Pre-flight: confirm the stream is open before sending
// (control streams expose closed state; check before each send)
if control_stream.is_closed() {
    // mark worker disconnected and skip/retry
}

match inject_barrier(mutation).await {
    Err(e) if e.to_string().contains("failed to send request to") => {
        tracing::warn!("control stream broken, awaiting recovery: {e}");
        // barrier manager will retry after worker reconnects
    }
    other => other?,
}

When it happens

Trigger: `inject_barrier` calling `control_stream.send(...)` (via the request builder) and the underlying channel returns an error — stream closed by the worker, connection reset, or worker process exit mid-send.

Common situations: Worker crash or OOM-kill during barrier injection; network interruption between meta and compute node; long-running barrier send across a node restart; madsim/fault-injection tests simulating stream failure.

Related errors

• bigquery insert error: end of resp stream
• Internal
• Cannot open client to compute node {addr:?}
• RpcError
• unconnected worker node {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b1266b28bc0c31ca. Report an issue: GitHub.