risingwavelabs/risingwave · warning

cluster under recovery[{}]

Error message

cluster under recovery[{}]

What it means

While the cluster is recovering, the barrier manager's recovery loop answers GetBackfillProgress requests immediately with `cluster under recovery[<reason>]` instead of processing them, because progress state is not available until recovery completes. Callers waiting on the oneshot channel receive this MetaError.

Source

Thrown at src/meta/src/barrier/worker.rs:1369

        let mut recover_txs = vec![];
        let mut update_barrier_requests = vec![];
        pin_mut!(recovery_future);
        let mut request_rx_closed = false;
        let new_state = loop {
            select! {
                biased;
                new_state = &mut recovery_future => {
                    break new_state.expect("Retry until recovery success.");
                }
                request = pin!(self.request_rx.recv()), if !request_rx_closed => {
                    let Some(request) = request else {
                        warn!("request rx channel closed during recovery");
                        request_rx_closed = true;
                        continue;
                    };
                    match request {
                        BarrierManagerRequest::GetBackfillProgress(tx) => {
                            let _ = tx.send(Err(anyhow!("cluster under recovery[{}]", recovery_reason).into()));
                        }
                        BarrierManagerRequest::GetFragmentBackfillProgress(tx) => {
                            let _ = tx.send(Err(anyhow!("cluster under recovery[{}]", recovery_reason).into()));
                        }
                        BarrierManagerRequest::GetCdcProgress(tx) => {
                            let _ = tx.send(Err(anyhow!("cluster under recovery[{}]", recovery_reason).into()));
                        }
                        BarrierManagerRequest::AdhocRecovery(tx) => {
                            recover_txs.push(tx);
                        }
                        BarrierManagerRequest::UpdateDatabaseBarrier(request) => {
                            update_barrier_requests.push(request);
                        }
                    }
                }
            }
        };

View on GitHub (pinned to 6469eb736d)

Solutions

1. Wait for recovery to finish (cluster becomes healthy) and retry the progress query.
2. Check the `<reason>` in brackets and the meta logs to understand what triggered recovery.
3. Add retry/backoff around progress polling so transient recovery windows don't surface as user-visible errors.
4. If recovery is stuck, inspect meta node logs and consider manual recovery (`ALTER SYSTEM ... ` / restart with recovery).

Example fix

// before: single-shot progress query
let p = client.get_backfill_progress(job_id).await?;
// after: tolerate recovery window
let p = loop {
    match client.get_backfill_progress(job_id).await {
        Ok(p) => break p,
        Err(e) if is_under_recovery(&e) => { tokio::time::sleep(Duration::from_secs(2)).await; }
        Err(e) => return Err(e),
    }
};

// only poll progress when the cluster is healthy
if !client.is_cluster_healthy().await { return; }

// retry through the recovery window with backoff
loop {
    match client.get_backfill_progress(job_id).await {
        Ok(p) => break Ok(p),
        Err(e) if is_under_recovery(&e) => sleep(BACKOFF).await,
        Err(e) => break Err(e),
    }
}

When it happens

Trigger: A frontend/client issues a query for backfill progress (GetBackfillProgress) while recovery_inner is replaying/awaiting barriers after a failover or manual recovery, with `recovery_reason` describing the trigger.

Common situations: Monitoring dashboards polling backfill progress during a crash-recovery window; clients reconnecting right after failover before the barrier manager is healthy.

Related errors

• recovered snapshot backfill job {} has no snapshot backfill
• recovered snapshot backfill job {} to upstream {} has not se
• snapshot epoch {} to upstream {} different to snapshot epoch
• snapshot backfill job {} has not set snapshot epoch
• since_timestamp requires at least one upstream table

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/20d802d131a34a9e. Report an issue: GitHub.