risingwavelabs/risingwave · error · anyhow

partial graph suspended

Error message

partial graph suspended

What it means

A TakeReceiver (exchange channel creation) request arrived while the target partial graph is in the Suspended state (typically after a scored stream error paused the graph pending a reset decision). The barrier worker cannot create the receiver and sends this error back to the remote requester.

Solutions

1. Wait for the partial graph to be reset/restarted and retry the TakeReceiver request with the new term id.
2. Inspect the root error that caused the suspension (reported via ResetPartialGraph response / ScoredStreamError).
3. Fix the underlying stream error (often a data or executor failure) that led to suspension.
4. Check that the meta node stops routing exchange requests to suspended graphs promptly.

Example fix

// before: buffering requests into a suspended graph
handle_actor_op(TakeReceiver { partial_graph_id, .. });
// after: check status first and retry after reset
if matches!(status, PartialGraphStatus::Resetting | PartialGraphStatus::Suspended(_)) {
    retry_after_reset(partial_graph_id, request);
}

// skip sending when graph is known suspended/resetting
if matches!(graph_status(partial_graph_id), Suspended(_) | Resetting) { defer_request(request); }

fn is_suspended(err: &anyhow::Error) -> bool { err.to_string() == "partial graph suspended" }

// queue and replay after reset completes
match result {
    Err(e) if is_suspended(&e) || is_resetting(&e) => pending_queue.push(request),
    other => other,
}

When it happens

Trigger: handle_actor_op matches PartialGraphStatus::Suspended(_) while processing LocalActorOperation::TakeReceiver for that partial graph id.

Common situations: Upstream actors keep requesting exchange channels while the downstream partial graph was suspended due to a stream error; recovery orchestration delay between suspend and reset.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• partial graph resetting
• take receiver on unmatched partial graph term to current…
• actor exited unexpectedly
• anyhow!(message.to_owned())
• barrier reader closed unexpectedly

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/2b9b701e40212096. Report an issue: GitHub.

Appendix: source

Thrown at src/stream/src/task/barrier_worker/mod.rs:551

                                    "take receiver on unmatched partial graph term"
                                );
                                anyhow!(
                                    "take receiver {:?} on unmatched partial graph term {} to current term {}",
                                    ids,
                                    term_id,
                                    graph.local_barrier_manager.term_id
                                )
                            } else {
                                let (upstream_actor_id, actor_id) = ids;
                                graph.new_actor_output_request(
                                    actor_id,
                                    upstream_actor_id,
                                    request,
                                );
                                return;
                            }
                        }
                        PartialGraphStatus::Suspended(_) => anyhow!("partial graph suspended"),
                        PartialGraphStatus::Resetting => anyhow!("partial graph resetting"),
                        PartialGraphStatus::Unspecified => unreachable!(),
                    },
                    Entry::Vacant(entry) => {
                        entry.insert(PartialGraphStatus::ReceivedExchangeRequest(vec![(
                            term_id, ids, request,
                        )]));
                        return;
                    }
                };
                if let TakeReceiverRequest::Remote { result_sender, .. } = request {
                    let _ = result_sender.send(Err(err.into()));
                }
            }
            #[cfg(test)]
            LocalActorOperation::GetCurrentLocalBarrierManager(sender) => {
                let partial_graph_status = self
                    .state

View on GitHub (pinned to 6469eb736d)