failed to send shutdown signal for streaming job

Error message

failed to send shutdown signal for streaming job {}: receiver dropped

What it means

In `cancel_jobs`, the meta node sends a shutdown oneshot sender (`tx`) over the job shutdown channel to the component that owns streaming job lifecycle. If that channel's receiver has been dropped, the send fails and canceling the job returns this error, meaning the shutdown request could not be delivered.

Solutions

Retry the cancellation once the meta node is healthy; verify with SHOW MATERIALIZED VIEWS / catalog whether the job was actually dropped.
Check meta node logs for shutdown task crashes and restart the meta service if the shutdown channel is permanently broken.
If the job is already gone but the error surfaces, treat it as benign; ensure cleanup paths tolerate `send` failure instead of failing the cancel.

Example fix

// before
Err(_) => return Err(anyhow::anyhow!("failed to send shutdown signal for streaming job {}: receiver dropped", job_id).into()),
// after: treat dropped receiver as job already shut down
Err(_) => {
    tracing::warn!("shutdown receiver for job {} dropped; assuming already stopped", job_id);
}

Defensive patterns

Strategy: retry

Validate before calling

// Check the shutdown channel is alive before cancelling
if shutdown_rx.is_closed() {
    return Err("shutdown receiver unavailable; meta may be shutting down");
}

Try / catch

match cancel_streaming_job(job_id).await {
    Err(e) if e.to_string().contains("receiver dropped") => {
        // receiver gone: verify job state, then retry after meta recovers
        if job_still_exists(job_id).await { retry_cancel(job_id).await?; }
    }
    other => other?,
}

Prevention

Avoid cancelling streaming jobs during meta node shutdown/restart
Monitor meta logs for shutdown task crashes
Make cancellation idempotent: check job existence before and after cancel

When it happens

Trigger: Calling cancel/drop for a streaming job while the shutdown signal receiver (held by the stream manager/actor shutdown task) is no longer alive — e.g. the shutdown worker already exited, meta node is shutting down, or the receiver task crashed.

Common situations: Dropping a MV/table while the meta node is concurrently shutting down or restarting; internal crashes of the shutdown handling task; heavy load causing the receiver task to be cancelled.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/fcef6a0294469d05. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/stream/stream_manager.rs:222

    }

    async fn cancel_jobs(
        &self,
        job_ids: Vec<JobId>,
    ) -> MetaResult<(HashMap<JobId, oneshot::Receiver<bool>>, Vec<JobId>)> {
        let mut jobs = self.streaming_jobs.lock().await;
        let mut receivers = HashMap::new();
        let mut background_job_ids = vec![];
        for job_id in job_ids {
            if let Some(job) = jobs.get_mut(&job_id) {
                if let Some(shutdown_tx) = job.shutdown_tx.take() {
                    let (tx, rx) = oneshot::channel();
                    match shutdown_tx.send(tx) {
                        Ok(()) => {
                            receivers.insert(job_id, rx);
                        }
                        Err(_) => {
                            return Err(anyhow::anyhow!(
                                "failed to send shutdown signal for streaming job {}: receiver dropped",
                                job_id
                            )
                            .into());
                        }
                    }
                }
            } else {
                // If these job ids do not exist in streaming_jobs, they should be background creating jobs.
                background_job_ids.push(job_id);
            }
        }

        Ok((receivers, background_job_ids))
    }
}

type CreatingStreamingJobInfoRef = Arc<CreatingStreamingJobInfo>;

View on GitHub (pinned to 6469eb736d)