risingwavelabs/risingwave · warning · BatchError

Receive shutdown msg: {msg:?}

Error message

Receive shutdown msg: {msg:?}

What it means

The batch task's `ShutdownMonitor::check` found that a shutdown/cancellation message (other than `Init`) has been recorded — e.g. the task was cancelled, aborted, or the system is shutting down — and bails with "Receive shutdown msg: {msg:?}". Join executors call this between poll steps to abort promptly instead of continuing to produce output for a task that is being torn down.

Source

Thrown at src/batch/src/task/task_execution.rs:271

pub struct ShutdownToken(tokio::sync::watch::Receiver<ShutdownMsg>);

impl ShutdownToken {
    /// Create an empty token.
    pub fn empty() -> Self {
        Self::new().1
    }

    /// Create a new token.
    pub fn new() -> (ShutdownSender, Self) {
        let (tx, rx) = tokio::sync::watch::channel(ShutdownMsg::Init);
        (ShutdownSender(tx), ShutdownToken(rx))
    }

    /// Return error if the shutdown token has been triggered.
    pub fn check(&self) -> Result<()> {
        match &*self.0.borrow() {
            ShutdownMsg::Init => Ok(()),
            msg => bail!("Receive shutdown msg: {msg:?}"),
        }
    }

    /// Wait until cancellation is requested.
    ///
    /// # Cancel safety
    /// This method is cancel safe.
    pub async fn cancelled(&mut self) {
        if matches!(*self.0.borrow(), ShutdownMsg::Init)
            && let Err(_err) = self.0.changed().await
        {
            std::future::pending::<()>().await;
        }
    }

    /// Return true if the shutdown token has been triggered.
    pub fn is_cancelled(&self) -> bool {
        !matches!(*self.0.borrow(), ShutdownMsg::Init)

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check whether the query was cancelled (user CANCEL QUERY, timeout, or `error while restarting`-style shutdown); if so this error is expected and can be ignored.
2. Look for sibling task/stage failures around the same timestamp — the task manager aborts all tasks of a failing stage, making this error secondary.
3. If it fires spuriously, inspect shutdown-token lifecycle in the task environment (premature `request_shutdown` or reused task env).
4. Retry the query if it was killed by cluster shutdown or failover.

// Callers treat shutdown as cancellation: map it to a cancelled/aborted error, not a data error
if let Err(e) = monitor.check() {
    tracing::info!("task aborted during join: {e}");
    return Err(e.into()); // task manager already knows; do not retry within the task
}

When it happens

Trigger: Raised by `ShutdownMonitor::check` (src/batch/src/task/task_execution.rs:271), called from the join executor polling loops (`do_inner_join`, `do_left_outer_join`, `do_left_semi_anti_join`, `do_right_outer_join`, `do_right_semi_anti_join`, `do_full_outer_join`). It fires when `ShutdownMsg` has transitioned from `Init` — via `request_shutdown`/cancellation token triggered by the task manager (query cancelled, task aborted, epoch timeout) or cluster shutdown — and the join loop next calls `check()`.

Common situations: A user cancels a long-running query in psql; the frontend/meta node aborts a stage (e.g. one node failed, killing sibling tasks); the batch task hits an epoch/time limit; RisingWave cluster shutdown while a join is mid-scan. Usually expected behavior during cancellation, but appearing without user cancellation suggests the task manager is aborting the stage (check for a sibling task failure).

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• {0}
• query_epoch not set in distributed lookup join
• Storage error: {0}
• Array error: {0}
• Expr error: {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b598601d31258c23. Report an issue: GitHub.