{"record":{"id":"68358de09c7a152e","repo":"risingwavelabs/risingwave","slug":"failed-to-get-back-pressure","errorCode":null,"errorMessage":"Failed to get back pressure","messagePattern":"Failed to get back pressure","errorType":"http","errorClass":"DashboardError","httpStatus":500,"severity":"error","filePath":"src/meta/src/dashboard/mod.rs","lineNumber":725,"sourceCode":"\n        let mut futures = Vec::new();\n\n        for worker_node in worker_nodes {\n            let client = srv.monitor_clients.get(&worker_node).await.map_err(err)?;\n            let client = Arc::new(client);\n            let fut = async move {\n                let result = client.get_streaming_stats().await.map_err(err)?;\n                Ok::<_, DashboardError>(result)\n            };\n            futures.push(fut);\n        }\n        let results = join_all(futures).await;\n\n        let mut all = GetStreamingStatsResponse::default();\n\n        for result in results {\n            let result = result\n                .map_err(|_| anyhow!(\"Failed to get back pressure\"))\n                .map_err(err)?;\n\n            // Aggregate fragment_stats\n            for (fragment_id, fragment_stats) in result.fragment_stats {\n                if let Some(s) = all.fragment_stats.get_mut(&fragment_id) {\n                    s.actor_count += fragment_stats.actor_count;\n                    s.current_epoch = min(s.current_epoch, fragment_stats.current_epoch);\n                } else {\n                    all.fragment_stats.insert(fragment_id, fragment_stats);\n                }\n            }\n\n            // Aggregate relation_stats\n            for (relation_id, relation_stats) in result.relation_stats {\n                if let Some(s) = all.relation_stats.get_mut(&relation_id) {\n                    s.actor_count += relation_stats.actor_count;\n                    s.current_epoch = min(s.current_epoch, relation_stats.current_epoch);\n                } else {","sourceCodeStart":707,"sourceCodeEnd":743,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/dashboard/mod.rs#L707-L743","documentation":"When the dashboard aggregates streaming stats, it fans out GetStreamingStats gRPC requests to all workers and joins the futures. If any individual worker's request fails (gRPC error, worker down, timeout), the whole aggregation is discarded and replaced with this generic 'Failed to get back pressure' error — the original cause is dropped by the `map_err(|_| ...)`.","triggerScenarios":"GET /api/v1/streaming_stats when at least one worker (compute node or meta) fails to respond to the GetStreamingStats RPC — e.g. the worker crashed, is restarting, or the gRPC channel is broken.","commonSituations":"Cluster during rolling upgrade; a compute node that was killed or OOMed; network partition between meta node and worker; worker still bootstrapping while the dashboard is polled.","solutions":["Check `SHOW NODES` / the dashboard worker list and confirm all workers are RUNNING; restart failed workers.","Inspect the meta node logs around the request for the underlying gRPC error that was swallowed.","Retry the dashboard request once the cluster is healthy; transient worker unavailability resolves this.","Improve the code: replace `map_err(|_| ...)` with a closure that preserves the source error for diagnosis."],"exampleFix":"// before\nresult.map_err(|_| anyhow!(\"Failed to get back pressure\")).map_err(err)?\n// after\nresult.map_err(|e| anyhow!(\"Failed to get back pressure: {e}\")).map_err(err)?","handlingStrategy":"retry","validationCode":"const health = await fetch('http://localhost:5691/api/v1/workers').then(r => r.json());\nif (health.some(w => w.state !== 'RUNNING')) throw new Error('Not all workers are RUNNING; fix cluster health before querying streaming stats');","typeGuard":null,"tryCatchPattern":"try { const stats = await getStreamingStats(); } catch (e) {\n  if (String(e).includes('Failed to get back pressure')) {\n    // check meta logs for the swallowed per-worker gRPC error, then retry with backoff\n  }\n}","preventionTips":["Monitor worker liveness before polling streaming stats endpoints.","Check meta node logs for the underlying gRPC failure hidden by the generic message.","Avoid scraping this endpoint during rolling upgrades or restarts."],"tags":["grpc","cluster-health","monitoring","risingwave"],"backgroundTag":"upstream-api-error","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}