risingwavelabs/risingwave · warning

The cluster is recovering-adhoc

Error message

The cluster is recovering-adhoc

What it means

check_status_running reports 'The cluster is recovering-adhoc' when the barrier manager is in Recovering(RecoveryReason::Adhoc) state — a recovery initiated ad-hoc (e.g. by operator command or internal trigger) rather than by bootstrap or failover. Commands requiring a Running barrier manager are rejected.

Source

Thrown at src/meta/src/barrier/manager.rs:175

    pub async fn get_hummock_version_id(&self) -> HummockVersionId {
        self.hummock_manager.get_version_id().await
    }
}

impl GlobalBarrierManager {
    /// Check the status of barrier manager, return error if it is not `Running`.
    pub fn check_status_running(&self) -> MetaResult<()> {
        let status = self.status.load();
        match &**status {
            BarrierManagerStatus::Starting
            | BarrierManagerStatus::Recovering(RecoveryReason::Bootstrap) => {
                bail!("The cluster is bootstrapping")
            }
            BarrierManagerStatus::Recovering(RecoveryReason::Failover(e)) => {
                Err(anyhow::anyhow!(e.clone()).context("The cluster is recovering"))?
            }
            BarrierManagerStatus::Recovering(RecoveryReason::Adhoc) => {
                bail!("The cluster is recovering-adhoc")
            }
            BarrierManagerStatus::Running => Ok(()),
        }
    }

    pub fn get_recovery_status(&self) -> PbRecoveryStatus {
        (&**self.status.load()).into()
    }
}

impl GlobalBarrierManager {
    #[expect(clippy::too_many_arguments)]
    pub async fn start(
        scheduled_barriers: schedule::ScheduledBarriers,
        env: MetaSrvEnv,
        metadata_manager: MetadataManager,
        hummock_manager: HummockManagerRef,
        serving_vnode_mapping: ServingVnodeMappingRef,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Wait until the ad-hoc recovery finishes before submitting new requests
2. Retry with backoff; the status will transition to Running
3. Avoid triggering ad-hoc recovery during live traffic windows
4. Monitor recovery status via get_recovery_status / metrics

if manager.get_recovery_status() == PbRecoveryStatus::Recovering { defer_requests(); }

if err.to_string() == "The cluster is recovering-adhoc" {
    retry_with_backoff(|| run_statement()).await?;
}

When it happens

Trigger: Issuing DDL/DML while an ad-hoc recovery is in progress (manually triggered recovery path).

Common situations: Operator-triggered recovery being exercised while clients remain active; automated tooling issuing recovery then immediately running queries.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• since_timestamp requires at least one upstream table
• The cluster is recovering
• since_timestamp epoch has not been resolved for snapshot bac
• The cluster is bootstrapping
• worker_id {} for actor {} does not exist

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/97fa7ff7c420109b. Report an issue: GitHub.