risingwavelabs/risingwave · warning

cluster under recovery

Error message

cluster under recovery[{}]

What it means

While the cluster is recovering, the barrier manager's recovery loop answers GetBackfillProgress requests immediately with `cluster under recovery[<reason>]` instead of processing them, because progress state is not available until recovery completes. Callers waiting on the oneshot channel receive this MetaError.

Solutions

  1. Wait for recovery to finish (cluster becomes healthy) and retry the progress query.
  2. Check the `<reason>` in brackets and the meta logs to understand what triggered recovery.
  3. Add retry/backoff around progress polling so transient recovery windows don't surface as user-visible errors.
  4. If recovery is stuck, inspect meta node logs and consider manual recovery (`ALTER SYSTEM ... ` / restart with recovery).

Example fix

// before: single-shot progress query
let p = client.get_backfill_progress(job_id).await?;
// after: tolerate recovery window
let p = loop {
    match client.get_backfill_progress(job_id).await {
        Ok(p) => break p,
        Err(e) if is_under_recovery(&e) => { tokio::time::sleep(Duration::from_secs(2)).await; }
        Err(e) => return Err(e),
    }
};
Defensive patterns

Strategy: retry

Validate before calling

// only poll progress when the cluster is healthy
if !client.is_cluster_healthy().await { return; }

Try / catch

// retry through the recovery window with backoff
loop {
    match client.get_backfill_progress(job_id).await {
        Ok(p) => break Ok(p),
        Err(e) if is_under_recovery(&e) => sleep(BACKOFF).await,
        Err(e) => break Err(e),
    }
}

Prevention

When it happens

Trigger: A frontend/client issues a query for backfill progress (GetBackfillProgress) while recovery_inner is replaying/awaiting barriers after a failover or manual recovery, with `recovery_reason` describing the trigger.

Common situations: Monitoring dashboards polling backfill progress during a crash-recovery window; clients reconnecting right after failover before the barrier manager is healthy.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/20d802d131a34a9e. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/barrier/worker.rs:1369

        let mut recover_txs = vec![];
        let mut update_barrier_requests = vec![];
        pin_mut!(recovery_future);
        let mut request_rx_closed = false;
        let new_state = loop {
            select! {
                biased;
                new_state = &mut recovery_future => {
                    break new_state.expect("Retry until recovery success.");
                }
                request = pin!(self.request_rx.recv()), if !request_rx_closed => {
                    let Some(request) = request else {
                        warn!("request rx channel closed during recovery");
                        request_rx_closed = true;
                        continue;
                    };
                    match request {
                        BarrierManagerRequest::GetBackfillProgress(tx) => {
                            let _ = tx.send(Err(anyhow!("cluster under recovery[{}]", recovery_reason).into()));
                        }
                        BarrierManagerRequest::GetFragmentBackfillProgress(tx) => {
                            let _ = tx.send(Err(anyhow!("cluster under recovery[{}]", recovery_reason).into()));
                        }
                        BarrierManagerRequest::GetCdcProgress(tx) => {
                            let _ = tx.send(Err(anyhow!("cluster under recovery[{}]", recovery_reason).into()));
                        }
                        BarrierManagerRequest::AdhocRecovery(tx) => {
                            recover_txs.push(tx);
                        }
                        BarrierManagerRequest::UpdateDatabaseBarrier(request) => {
                            update_barrier_requests.push(request);
                        }
                    }
                }
            }
        };

View on GitHub (pinned to 6469eb736d)