risingwavelabs/risingwave · error · ConnectorError

all senders are dropped

Error message

all senders are dropped

What it means

`into_data_stream` multiplexes split message streams into channel senders. If every sender (all split channels) has been dropped before completion, the select loop can never receive new messages, so it bails with "all senders are dropped" instead of hanging forever. This usually reflects a lifecycle bug or upstream cancellation rather than user data.

Source

Thrown at src/connector/src/source/cdc/source/reader.rs:247

            match result {
                Ok(GetEventStreamResponse { events, .. }) => {
                    tracing::trace!("receive {} cdc events ", events.len());
                    let msgs = events.into_iter().map(SourceMessage::from).collect_vec();
                    yield msgs;
                }
                Err(e) => {
                    GLOBAL_ERROR_METRICS.user_source_error.report([
                        "cdc_source".to_owned(),
                        source_id.clone(),
                        self.source_ctx.source_name.clone(),
                        self.source_ctx.fragment_id.to_string(),
                    ]);
                    Err(e)?;
                }
            }
        }

        bail!("all senders are dropped");
    }
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect downstream executor logs for panics or cancellation that dropped the receivers, and fix that root cause.
2. Restart the source/actor to rebuild channels; the error itself is unrecoverable by retry.
3. If it happens reproducibly, report/fix in reader.rs:247 — the loop should detect sender closure per-channel and only bail when all splits finished.

Example fix

// before
bail!("all senders are dropped");
// after
break; // treat dropped senders as stream end if all splits completed normally

match SourceReader::new(...).into_stream().await {
    Ok(stream) => ..., // consume with backpressure-safe downstream
    Err(e) if e.to_string().contains("all senders are dropped") => {
        tracing::error!("downstream channels closed; check downstream executor logs");
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: All downstream receivers for the split streams are dropped (downstream actor failure/cancellation, barrier handler dropped channels, source executor being terminated) while `into_stream`/`into_data_stream` still waits for data.

Common situations: A downstream actor panicked or the stream chunk channel was closed due to backpressure/kill; cluster scale-in terminating executors; internal bug where channels are closed before the EOF sentinel is sent.

Related errors

• anyhow!(err_msg)
• Unexpected node body for stream cdc scan
• no chunk in IngestDmlPayloadRequest
• broken broadcast_channel
• broken hash_shuffle_channel

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/51930898534ae1f4. Report an issue: GitHub.