{"record":{"id":"20d802d131a34a9e","repo":"risingwavelabs/risingwave","slug":"cluster-under-recovery","errorCode":null,"errorMessage":"cluster under recovery[{}]","messagePattern":"cluster under recovery\\[(.+?)\\]","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/meta/src/barrier/worker.rs","lineNumber":1369,"sourceCode":"        let mut recover_txs = vec![];\n        let mut update_barrier_requests = vec![];\n        pin_mut!(recovery_future);\n        let mut request_rx_closed = false;\n        let new_state = loop {\n            select! {\n                biased;\n                new_state = &mut recovery_future => {\n                    break new_state.expect(\"Retry until recovery success.\");\n                }\n                request = pin!(self.request_rx.recv()), if !request_rx_closed => {\n                    let Some(request) = request else {\n                        warn!(\"request rx channel closed during recovery\");\n                        request_rx_closed = true;\n                        continue;\n                    };\n                    match request {\n                        BarrierManagerRequest::GetBackfillProgress(tx) => {\n                            let _ = tx.send(Err(anyhow!(\"cluster under recovery[{}]\", recovery_reason).into()));\n                        }\n                        BarrierManagerRequest::GetFragmentBackfillProgress(tx) => {\n                            let _ = tx.send(Err(anyhow!(\"cluster under recovery[{}]\", recovery_reason).into()));\n                        }\n                        BarrierManagerRequest::GetCdcProgress(tx) => {\n                            let _ = tx.send(Err(anyhow!(\"cluster under recovery[{}]\", recovery_reason).into()));\n                        }\n                        BarrierManagerRequest::AdhocRecovery(tx) => {\n                            recover_txs.push(tx);\n                        }\n                        BarrierManagerRequest::UpdateDatabaseBarrier(request) => {\n                            update_barrier_requests.push(request);\n                        }\n                    }\n                }\n            }\n        };\n","sourceCodeStart":1351,"sourceCodeEnd":1387,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/barrier/worker.rs#L1351-L1387","documentation":"While the cluster is recovering, the barrier manager's recovery loop answers GetBackfillProgress requests immediately with `cluster under recovery[<reason>]` instead of processing them, because progress state is not available until recovery completes. Callers waiting on the oneshot channel receive this MetaError.","triggerScenarios":"A frontend/client issues a query for backfill progress (GetBackfillProgress) while recovery_inner is replaying/awaiting barriers after a failover or manual recovery, with `recovery_reason` describing the trigger.","commonSituations":"Monitoring dashboards polling backfill progress during a crash-recovery window; clients reconnecting right after failover before the barrier manager is healthy.","solutions":["Wait for recovery to finish (cluster becomes healthy) and retry the progress query.","Check the `<reason>` in brackets and the meta logs to understand what triggered recovery.","Add retry/backoff around progress polling so transient recovery windows don't surface as user-visible errors.","If recovery is stuck, inspect meta node logs and consider manual recovery (`ALTER SYSTEM ... ` / restart with recovery)."],"exampleFix":"// before: single-shot progress query\nlet p = client.get_backfill_progress(job_id).await?;\n// after: tolerate recovery window\nlet p = loop {\n    match client.get_backfill_progress(job_id).await {\n        Ok(p) => break p,\n        Err(e) if is_under_recovery(&e) => { tokio::time::sleep(Duration::from_secs(2)).await; }\n        Err(e) => return Err(e),\n    }\n};","handlingStrategy":"retry","validationCode":"// only poll progress when the cluster is healthy\nif !client.is_cluster_healthy().await { return; }","typeGuard":null,"tryCatchPattern":"// retry through the recovery window with backoff\nloop {\n    match client.get_backfill_progress(job_id).await {\n        Ok(p) => break Ok(p),\n        Err(e) if is_under_recovery(&e) => sleep(BACKOFF).await,\n        Err(e) => break Err(e),\n    }\n}","preventionTips":["Add backoff/retry to all progress-polling jobs.","Gate dashboards on cluster health status.","Expect this error during every failover; treat it as transient, not fatal."],"tags":["meta","recovery","backfill","transient"],"backgroundTag":"cluster-under-recovery","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}