risingwavelabs/risingwave · error

non-empty

Error message

non-empty

What it means

An `.expect("non-empty")` panic in `next_to_commit`: after the backoff wait resolves, the code assumes `prepared_epochs` is non-empty and takes its front element. The invariant is that the backoff timer is only armed when a prepared epoch exists; if it fires after that epoch was removed, the invariant is broken.

Solutions

1. Re-check `prepared_epochs` after the backoff wait and loop/re-arm instead of blindly taking the front element.
2. Ensure `ack_committed`/`failed_committed` clear or recompute `backoff_state` whenever `prepared_epochs` changes.
3. Inspect logs for interleaved ack/commit of epochs racing with backoff waits and reproduce to fix the race.
4. Replace `.expect("non-empty")` with an early `return`/continue when the queue is empty.

Example fix

// before
let item = self.prepared_epochs.front().cloned().expect("non-empty");
return Ok(item);
// after
match self.prepared_epochs.front().cloned() {
    Some(item) => return Ok(item),
    None => { self.backoff_state = None; continue; } // timer fired for a consumed epoch
}

// check queue before acting on the backoff timer
if self.prepared_epochs.is_empty() { self.backoff_state = None; return Ok(()); }

fn has_prepared(q: &VecDeque<(u64, _, _)>) -> bool { !q.is_empty() }

// panic-based: cannot catch; restructure the select so the backoff branch re-checks the queue
if self.prepared_epochs.front().is_none() { continue; }

When it happens

Trigger: The backoff future fires after `prepared_epochs` was drained by `ack_committed`/`failed_committed` or by the `job_committed_epoch_rx` branch — a race between the select arms or a bug leaving the backoff state armed for a consumed epoch.

Common situations: High-frequency commit/ack churn where backoff timers overlap epoch transitions; meta-node task scheduling delays letting a stale timer win the select; bugs in resetting `backoff_state` when epochs are consumed.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Creating such a sink will result in circular dependency.
• Dropping sink into table is not allowed for unmigrated table
• duplicate vnode . request vnode: , prev vnode: . pending…
• expected exactly one sink fragment for each sink, but got
• failed to parse relation definition

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/39e6f01c77a28cde. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/manager/sink_coordination/coordinator_worker.rs:179

            .map(jitter)
            .map(|delay| Box::pin(tokio::time::sleep(delay)))
    }

    async fn next_to_commit(
        &mut self,
    ) -> anyhow::Result<(u64, Option<Vec<u8>>, Option<PbSinkSchemaChange>)> {
        loop {
            let wait_backoff = async {
                if self.prepared_epochs.is_empty() {
                    pending::<()>().await;
                } else if let Some((backoff_fut, _)) = &mut self.backoff_state {
                    backoff_fut.await;
                }
            };

            select! {
                _ = wait_backoff => {
                    let item = self.prepared_epochs.front().cloned().expect("non-empty");
                    return Ok(item);
                }

                recv_epoch = self.job_committed_epoch_rx.recv() => {
                    let Some(recv_epoch) = recv_epoch else {
                        return Err(anyhow!(
                            "Hummock committed epoch sender closed unexpectedly"
                        ));
                    };
                    self.curr_hummock_committed_epoch = recv_epoch;
                    while let Some((epoch, metadata, schema_change)) = self.pending_epochs.pop_front_if(|(epoch, _, _)| *epoch <= recv_epoch) {
                        if let Some((last_epoch, _, _)) = self.prepared_epochs.back() {
                            assert!(epoch > *last_epoch, "prepared epochs must be in increasing order");
                        }
                        self.prepared_epochs.push_back((epoch, metadata, schema_change));
                    }
                }
            }

View on GitHub (pinned to 6469eb736d)