{"record":{"id":"dd032c83a705f74b","repo":"risingwavelabs/risingwave","slug":"the-cluster-is-recovering","errorCode":null,"errorMessage":"The cluster is recovering","messagePattern":"The cluster is recovering","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/meta/src/barrier/manager.rs","lineNumber":172,"sourceCode":"        Ok(())\n    }\n\n    pub async fn get_hummock_version_id(&self) -> HummockVersionId {\n        self.hummock_manager.get_version_id().await\n    }\n}\n\nimpl GlobalBarrierManager {\n    /// Check the status of barrier manager, return error if it is not `Running`.\n    pub fn check_status_running(&self) -> MetaResult<()> {\n        let status = self.status.load();\n        match &**status {\n            BarrierManagerStatus::Starting\n            | BarrierManagerStatus::Recovering(RecoveryReason::Bootstrap) => {\n                bail!(\"The cluster is bootstrapping\")\n            }\n            BarrierManagerStatus::Recovering(RecoveryReason::Failover(e)) => {\n                Err(anyhow::anyhow!(e.clone()).context(\"The cluster is recovering\"))?\n            }\n            BarrierManagerStatus::Recovering(RecoveryReason::Adhoc) => {\n                bail!(\"The cluster is recovering-adhoc\")\n            }\n            BarrierManagerStatus::Running => Ok(()),\n        }\n    }\n\n    pub fn get_recovery_status(&self) -> PbRecoveryStatus {\n        (&**self.status.load()).into()\n    }\n}\n\nimpl GlobalBarrierManager {\n    #[expect(clippy::too_many_arguments)]\n    pub async fn start(\n        scheduled_barriers: schedule::ScheduledBarriers,\n        env: MetaSrvEnv,","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/manager.rs#L154-L190","documentation":"check_status_running reports 'The cluster is recovering' when the barrier manager is in Recovering(RecoveryReason::Failover(e)) state. The inner error `e` is the original failover cause; it is wrapped with context 'The cluster is recovering'. Requests are rejected until recovery completes.","triggerScenarios":"Sending DDL/DML while the cluster is recovering from a failover (e.g. meta node restart or worker failure), where the recovery was triggered with a Failover reason carrying the original error.","commonSituations":"Queries issued during meta failover; workload automation not pausing during recovery; monitoring probes surfacing the underlying recovery cause.","solutions":["Wait for recovery completion before retrying (cluster returns to Running)","Inspect the wrapped inner error for the original failover cause and address it if recovery is stuck","Add retry logic that treats this as transient","Reduce failover frequency by fixing the root instability (worker OOM, network, etc.)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"if let Some(e) = manager.get_recovery_status().and_then(check_failover_reason) {\n    log::warn!(\"cluster recovering due to: {}\", e);\n}","typeGuard":null,"tryCatchPattern":"let result = run_statement().await;\nif let Err(e) = &result {\n    if e.chain().any(|c| c.to_string() == \"The cluster is recovering\") {\n        retry_with_backoff(|| run_statement()).await?;\n    }\n}","preventionTips":["Pause traffic during planned failovers","Use retry-with-backoff middleware for meta requests","Monitor recovery duration and the wrapped failover cause"],"tags":["rust","meta","barrier","recovery","failover"],"backgroundTag":"invalid-state-transition","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}