risingwavelabs/risingwave · error

Hummock committed epoch sender closed unexpectedly

Error message

Hummock committed epoch sender closed unexpectedly

What it means

Returned by `next_to_commit` when the `job_committed_epoch_rx` channel (carrying Hummock committed epochs from the observer manager) yields `None`, meaning all senders were dropped. Without a live committed-epoch feed the sink coordinator cannot make progress and errors out instead of hanging.

Source

Thrown at src/meta/src/manager/sink_coordination/coordinator_worker.rs:185

    ) -> anyhow::Result<(u64, Option<Vec<u8>>, Option<PbSinkSchemaChange>)> {
        loop {
            let wait_backoff = async {
                if self.prepared_epochs.is_empty() {
                    pending::<()>().await;
                } else if let Some((backoff_fut, _)) = &mut self.backoff_state {
                    backoff_fut.await;
                }
            };

            select! {
                _ = wait_backoff => {
                    let item = self.prepared_epochs.front().cloned().expect("non-empty");
                    return Ok(item);
                }

                recv_epoch = self.job_committed_epoch_rx.recv() => {
                    let Some(recv_epoch) = recv_epoch else {
                        return Err(anyhow!(
                            "Hummock committed epoch sender closed unexpectedly"
                        ));
                    };
                    self.curr_hummock_committed_epoch = recv_epoch;
                    while let Some((epoch, metadata, schema_change)) = self.pending_epochs.pop_front_if(|(epoch, _, _)| *epoch <= recv_epoch) {
                        if let Some((last_epoch, _, _)) = self.prepared_epochs.back() {
                            assert!(epoch > *last_epoch, "prepared epochs must be in increasing order");
                        }
                        self.prepared_epochs.push_back((epoch, metadata, schema_change));
                    }
                }
            }
        }
    }

    fn push_new_item(
        &mut self,
        epoch: u64,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check meta-node logs for the shutdown/crash of the component feeding job_committed_epoch (observer manager / hummock manager).
2. Ensure the coordinator worker task is cancelled before its epoch-feed sender is dropped, in the right shutdown order.
3. If the sender task crashed, find and fix its panic, then restart the meta node.
4. Verify the sink coordinator is only started after the committed-epoch channel is wired up.

// ensure the committed-epoch feed is alive before running the coordinator loop
if job_committed_epoch_tx.is_closed() { return Err(anyhow!("committed epoch feed not started")); }

match worker.next_to_commit().await {
    Err(e) if e.to_string().contains("sender closed") => {
        tracing::warn!("hummock committed epoch feed closed; stopping sink coordinator");
        // trigger orderly shutdown instead of retrying
    }
    other => other?,
}

When it happens

Trigger: The sender side of `job_committed_epoch_rx` (e.g. `observe_committed_epoch` feed in the meta node) is closed or dropped before/while `next_to_commit` polls it — typically when the meta service shuts down or the observer task terminates.

Common situations: Meta-node graceful shutdown while a sink coordinator is still running; a panic or early return in the task holding the sender; misconfigured startup where the committed-epoch observer never spawns.

Related errors

• unable to send response: {:?}
• channel closed
• Lance fragment write task stopped before accepting a record
• (dynamic: channel send error on shutdown signal)
• replace sink must not use snapshot backfill

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/06c031211b6890c6. Report an issue: GitHub.