risingwavelabs/risingwave · critical

infinite

Error message

infinite

What it means

This panic fires in `failed_committed` when the sink coordinator needs a backoff delay before retrying a failed epoch commit, and `strategy.next()` on the retry backoff strategy returns `None`. `Backoff::next()` only returns `None` once the iterator is exhausted, which for a properly configured `Backoff` (e.g. `Backoff::exponential` with `max_attempts` unset / infinite) never happens. The `.expect("infinite")` encodes the invariant that the meta-node must retry failed commits forever, so exhausting the strategy is an internal bug in how the backoff strategy was constructed.

Source

Thrown at src/meta/src/manager/sink_coordination/coordinator_worker.rs:242

            self.prepared_epochs
                .push_back((epoch, metadata, schema_change));
        }
    }

    async fn ack_committed(&mut self, epoch: u64) -> anyhow::Result<()> {
        self.backoff_state = None;
        let (last_epoch, _, _) = self.prepared_epochs.pop_front().expect("non-empty");
        assert_eq!(last_epoch, epoch);

        commit_and_prune_epoch(&self.db, self.sink_id, epoch, self.last_committed_epoch).await?;
        self.last_committed_epoch = Some(epoch);
        Ok(())
    }

    fn failed_committed(&mut self, epoch: u64, err: SinkError) {
        assert_eq!(self.prepared_epochs.front().expect("non-empty").0, epoch,);
        if let Some((prev_fut, strategy)) = &mut self.backoff_state {
            let new_fut = strategy.next().expect("infinite");
            *prev_fut = new_fut;
        } else {
            let mut strategy = Self::get_retry_backoff_strategy();
            let backoff_fut = strategy.next().expect("infinite");
            self.backoff_state = Some((backoff_fut, strategy));
        }
        tracing::error!(
            error = %err.as_report(),
            %self.sink_id,
            "failed to commit epoch {}, Retrying after backoff",
            epoch,
        );
    }

    fn is_empty(&self) -> bool {
        self.pending_epochs.is_empty() && self.prepared_epochs.is_empty()
    }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Ensure `get_retry_backoff_strategy` builds an unbounded backoff (no `max_attempts`/finite attempt limit) so `next()` never returns `None`.
2. Check for local modifications or feature flags that swap the strategy for a bounded one in tests or config.
3. If retries should be bounded, replace the `.expect("infinite")` handling with graceful abort/failover logic instead of panicking.
4. While the panic persists, fix the underlying `SinkError` (check sink connectivity) and restart the meta node to reset `backoff_state`.

Example fix

// before
let mut strategy = Backoff::exponential(Duration::from_millis(100)).max_attempts(10);
let backoff_fut = strategy.next().expect("infinite");

// after
let mut strategy = Backoff::exponential(Duration::from_millis(100)); // unbounded
let backoff_fut = strategy.next().expect("infinite");

// Before deploying, assert the backoff strategy never exhausts
let mut s = CoordinatorWorker::get_retry_backoff_strategy();
for _ in 0..1_000_000 { assert!(s.next().is_some(), "backoff must be infinite"); }

// Cannot be caught at runtime (panic); guard construction:
fn get_retry_backoff_strategy() -> Backoff {
    Backoff::exponential(Duration::from_millis(100)) // no max_attempts
}

When it happens

Trigger: A sink commit of a prepared epoch fails with a `SinkError`, `failed_committed` is invoked, and the existing or freshly created `Backoff` strategy yields `None` from `next()` — only possible if the strategy was built with a finite attempt count (e.g. `max_attempts` set) or was fully consumed across retries.

Common situations: Someone changed `get_retry_backoff_strategy` to a bounded backoff (e.g. for testing) and shipped it; or a finite `max_attempts` was configured while downstream code still assumes endless retries; persistent sink (e.g. Kafka) outages that drive the retry loop through more attempts than the bound allows.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• should have been handled
• should exist
• IcebergSinkWriter should be initialized before barrier
• {connection_err (pulsar::Error) after retries exhausted}
• BatchPosixFsReader should not hit this branch. refer to `bat

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/e7e53974191b08ec. Report an issue: GitHub.