What causes "The cluster is recovering" in risingwavelabs/risingwave?

Wait for recovery completion before retrying (cluster returns to Running)

The cluster is recovering

Error message

The cluster is recovering

What it means

check_status_running reports 'The cluster is recovering' when the barrier manager is in Recovering(RecoveryReason::Failover(e)) state. The inner error `e` is the original failover cause; it is wrapped with context 'The cluster is recovering'. Requests are rejected until recovery completes.

Source

Thrown at src/meta/src/barrier/manager.rs:172

        Ok(())
    }

    pub async fn get_hummock_version_id(&self) -> HummockVersionId {
        self.hummock_manager.get_version_id().await
    }
}

impl GlobalBarrierManager {
    /// Check the status of barrier manager, return error if it is not `Running`.
    pub fn check_status_running(&self) -> MetaResult<()> {
        let status = self.status.load();
        match &**status {
            BarrierManagerStatus::Starting
            | BarrierManagerStatus::Recovering(RecoveryReason::Bootstrap) => {
                bail!("The cluster is bootstrapping")
            }
            BarrierManagerStatus::Recovering(RecoveryReason::Failover(e)) => {
                Err(anyhow::anyhow!(e.clone()).context("The cluster is recovering"))?
            }
            BarrierManagerStatus::Recovering(RecoveryReason::Adhoc) => {
                bail!("The cluster is recovering-adhoc")
            }
            BarrierManagerStatus::Running => Ok(()),
        }
    }

    pub fn get_recovery_status(&self) -> PbRecoveryStatus {
        (&**self.status.load()).into()
    }
}

impl GlobalBarrierManager {
    #[expect(clippy::too_many_arguments)]
    pub async fn start(
        scheduled_barriers: schedule::ScheduledBarriers,
        env: MetaSrvEnv,

View on GitHub (pinned to 6469eb736d)

Solutions

Wait for recovery completion before retrying (cluster returns to Running)
Inspect the wrapped inner error for the original failover cause and address it if recovery is stuck
Add retry logic that treats this as transient
Reduce failover frequency by fixing the root instability (worker OOM, network, etc.)

Defensive patterns

Strategy: retry

Validate before calling

if let Some(e) = manager.get_recovery_status().and_then(check_failover_reason) {
    log::warn!("cluster recovering due to: {}", e);
}

Try / catch

let result = run_statement().await;
if let Err(e) = &result {
    if e.chain().any(|c| c.to_string() == "The cluster is recovering") {
        retry_with_backoff(|| run_statement()).await?;
    }
}

Prevention

Pause traffic during planned failovers
Use retry-with-backoff middleware for meta requests
Monitor recovery duration and the wrapped failover cause

When it happens

Trigger: Sending DDL/DML while the cluster is recovering from a failover (e.g. meta node restart or worker failure), where the recovery was triggered with a Failover reason carrying the original error.

Common situations: Queries issued during meta failover; workload automation not pausing during recovery; monitoring probes surfacing the underlying recovery cause.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/dd032c83a705f74b. Report an issue: GitHub.