risingwavelabs/risingwave · error · StreamExecutorError

end of new output request

Error message

end of new output request

What it means

This anyhow error is raised in the DispatchExecutor's output collection when `new_output_request_rx.recv()` returns None, meaning all senders of the `new_output_request` mpsc channel have been dropped. The executor expected a new-output request from a downstream actor but the channel's sender side is gone, so no request can ever arrive. It indicates the actor/dispatcher lifecycle ended or was torn down unexpectedly while a downstream output resolution was still pending.

Source

Thrown at src/stream/src/executor/dispatch.rs:146

    ) -> StreamResult<Vec<Output>> {
        fn resolve_output(downstream_actor: ActorId, request: NewOutputRequest) -> Output {
            let tx = match request {
                NewOutputRequest::Local(tx) | NewOutputRequest::Remote(tx) => tx,
            };
            Output::new(downstream_actor, tx)
        }
        let mut outputs = Vec::with_capacity(downstream_actors.len());
        for &downstream_actor in downstream_actors {
            let output =
                if let Some(request) = self.pending_new_output_requests.remove(&downstream_actor) {
                    resolve_output(downstream_actor, request)
                } else {
                    loop {
                        let (requested_actor, request) = self
                            .new_output_request_rx
                            .recv()
                            .await
                            .ok_or_else(|| anyhow!("end of new output request"))?;
                        if requested_actor == downstream_actor {
                            break resolve_output(requested_actor, request);
                        } else {
                            assert!(
                                self.pending_new_output_requests
                                    .insert(requested_actor, request)
                                    .is_none(),
                                "duplicated inflight new output requests from actor {}",
                                requested_actor
                            );
                        }
                    }
                };
            outputs.push(output);
        }
        Ok(outputs)
    }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check upstream actor logs for early termination or failure that dropped the new_output_request senders.
2. Verify the actor graph/dispatcher wiring so every downstream that calls collect_outputs has a live upstream sender.
3. Ensure actor shutdown ordering tears down the dispatcher executor before dropping senders, so collect_outputs is not invoked on a dead channel.
4. Retry the rescale/job operation after the failed dispatcher update; this is a lifecycle race rather than a data error.

Example fix

// before: blindly resolving output even if channel is dead
let (requested_actor, request) = self.new_output_request_rx.recv().await.unwrap();
// after: propagate a descriptive error and handle channel closure
let (requested_actor, request) = self
    .new_output_request_rx
    .recv()
    .await
    .ok_or_else(|| anyhow!("end of new output request"))?;

// check channel liveness before collecting outputs
if self.new_output_request_tx.is_closed() {
    return Err(anyhow!("new_output_request channel closed before collect_outputs"));
}

match self.new_output_request_rx.recv().await {
    Some((actor, req)) => resolve_output(actor, req),
    None => {
        tracing::warn!("output request channel ended; aborting dispatcher update");
        return Err(anyhow!("end of new output request"));
    }
}

When it happens

Trigger: Calling `collect_outputs` (via `add_dispatchers` or `pre_update_dispatcher`) while the `new_output_request_tx` senders have all been dropped, so `recv()` returns None instead of a `(requested_actor, request)` message.

Common situations: Upstream actor terminated or failed before sending the output request; actor rescale/dispatcher update racing with actor shutdown; misconfigured actor graph where the expected sender was never created; stream job cancellation during dispatcher rebalance.

Related errors

• end of writer request stream
• next offset {:?} should be later than current offset {:?}
• new item epoch {} does not match current chunk offset epoch
• new item epoch {} does not exceed barrier offset epoch {}
• Division by zero

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/33f61224e5d3a511. Report an issue: GitHub.