{"record":{"id":"97fa7ff7c420109b","repo":"risingwavelabs/risingwave","slug":"the-cluster-is-recovering-adhoc","errorCode":null,"errorMessage":"The cluster is recovering-adhoc","messagePattern":"The cluster is recovering-adhoc","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/meta/src/barrier/manager.rs","lineNumber":175,"sourceCode":"    pub async fn get_hummock_version_id(&self) -> HummockVersionId {\n        self.hummock_manager.get_version_id().await\n    }\n}\n\nimpl GlobalBarrierManager {\n    /// Check the status of barrier manager, return error if it is not `Running`.\n    pub fn check_status_running(&self) -> MetaResult<()> {\n        let status = self.status.load();\n        match &**status {\n            BarrierManagerStatus::Starting\n            | BarrierManagerStatus::Recovering(RecoveryReason::Bootstrap) => {\n                bail!(\"The cluster is bootstrapping\")\n            }\n            BarrierManagerStatus::Recovering(RecoveryReason::Failover(e)) => {\n                Err(anyhow::anyhow!(e.clone()).context(\"The cluster is recovering\"))?\n            }\n            BarrierManagerStatus::Recovering(RecoveryReason::Adhoc) => {\n                bail!(\"The cluster is recovering-adhoc\")\n            }\n            BarrierManagerStatus::Running => Ok(()),\n        }\n    }\n\n    pub fn get_recovery_status(&self) -> PbRecoveryStatus {\n        (&**self.status.load()).into()\n    }\n}\n\nimpl GlobalBarrierManager {\n    #[expect(clippy::too_many_arguments)]\n    pub async fn start(\n        scheduled_barriers: schedule::ScheduledBarriers,\n        env: MetaSrvEnv,\n        metadata_manager: MetadataManager,\n        hummock_manager: HummockManagerRef,\n        serving_vnode_mapping: ServingVnodeMappingRef,","sourceCodeStart":157,"sourceCodeEnd":193,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/manager.rs#L157-L193","documentation":"check_status_running reports 'The cluster is recovering-adhoc' when the barrier manager is in Recovering(RecoveryReason::Adhoc) state — a recovery initiated ad-hoc (e.g. by operator command or internal trigger) rather than by bootstrap or failover. Commands requiring a Running barrier manager are rejected.","triggerScenarios":"Issuing DDL/DML while an ad-hoc recovery is in progress (manually triggered recovery path).","commonSituations":"Operator-triggered recovery being exercised while clients remain active; automated tooling issuing recovery then immediately running queries.","solutions":["Wait until the ad-hoc recovery finishes before submitting new requests","Retry with backoff; the status will transition to Running","Avoid triggering ad-hoc recovery during live traffic windows","Monitor recovery status via get_recovery_status / metrics"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if manager.get_recovery_status() == PbRecoveryStatus::Recovering { defer_requests(); }","typeGuard":null,"tryCatchPattern":"if err.to_string() == \"The cluster is recovering-adhoc\" {\n    retry_with_backoff(|| run_statement()).await?;\n}","preventionTips":["Schedule ad-hoc recoveries during maintenance windows","Gate client traffic on barrier-manager Running status","Track ad-hoc recovery events in observability dashboards"],"tags":["rust","meta","barrier","recovery"],"backgroundTag":"invalid-state-transition","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}