{"record":{"id":"b6dff520712ccba4","repo":"risingwavelabs/risingwave","slug":"trigger-manual-compaction-wait-report-failed-comp","errorCode":null,"errorMessage":"trigger_manual_compaction wait report failed. compaction_group {}","messagePattern":"trigger_manual_compaction wait report failed\\. compaction_group (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/hummock/manager/compaction/mod.rs","lineNumber":1158,"sourceCode":"        let report_rx = self.register_compaction_task_report_waiter(task_id);\n        if let Err(err) = compactor\n            .send_event(ResponseEvent::CompactTask(compact_task.into()))\n            .with_context(|| {\n                format!(\n                    \"Failed to trigger compaction task for compaction_group {}\",\n                    compaction_group,\n                )\n            })\n        {\n            self.remove_compaction_task_report_waiter(task_id);\n            return Err(err.into());\n        }\n\n        let report_result = match report_rx.await {\n            Ok(result) => result,\n            Err(_) => {\n                self.remove_compaction_task_report_waiter(task_id);\n                return Err(anyhow::anyhow!(\n                    \"trigger_manual_compaction wait report failed. compaction_group {}\",\n                    compaction_group\n                )\n                .into());\n            }\n        };\n        if !report_result.reported {\n            return Err(anyhow::anyhow!(\n                \"trigger_manual_compaction report not accepted. task_id {}\",\n                report_result.task_id\n            )\n            .into());\n        }\n\n        if report_result.task_status == TaskStatus::NoAvailCpuResourceCanceled\n            || report_result.task_status == TaskStatus::NoAvailMemoryResourceCanceled\n        {\n            return Ok(ManualCompactionTriggerResult::Retry);","sourceCodeStart":1140,"sourceCodeEnd":1176,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/hummock/manager/compaction/mod.rs#L1140-L1176","documentation":"After dispatching a manual compaction task, trigger_manual_compaction awaits a one-shot report channel (report_rx). If the sender is dropped without sending (the compactor never reported), the await returns Err and the waiter is removed, yielding this error.","triggerScenarios":"The compactor that received the task crashes, disconnects, or is cancelled before reporting; the report-waiter is dropped due to manager state cleanup; the oneshot channel closes without a send.","commonSituations":"Compactor node killed or OOMed mid-task; network partition between meta and compactor; meta failover during a long-running manual compaction; shutting down the cluster right after triggering compaction.","solutions":["Retry the manual compaction after confirming a compactor is healthy.","Check compactor logs/metrics for the task_id and why it exited without reporting.","Verify compactor-meta connectivity and that heartbeat/worker timeouts are not too aggressive."],"exampleFix":"// before\nlet result = report_rx.await.map_err(|_| anyhow!(\"trigger_manual_compaction wait report failed. compaction_group {}\", group))?;\n// after\n// add a timeout so a silent compactor is detected explicitly and the task is retried:\nlet result = tokio::time::timeout(Duration::from_secs(600), report_rx).await\n    .map_err(|_| anyhow!(\"compaction report timed out for group {}\", group))?\n    .map_err(|_| anyhow!(\"compactor dropped report channel for group {}\", group))?","handlingStrategy":"retry","validationCode":"// ensure at least one healthy compactor is registered before triggering\nassert!(compactor_heartbeats.iter().any(|h| h.recent()), \"no healthy compactor registered\");","typeGuard":null,"tryCatchPattern":"match trigger_manual_compaction(opt).await {\n    Err(e) if e.to_string().contains(\"wait report failed\") => {\n        tokio::time::sleep(backoff).await; // compactor died/crashed mid-task; retry after backoff\n        retry_with_backoff();\n    },\n    other => other?,\n}","preventionTips":["Monitor compactor health/heartbeats before triggering manual compaction","Avoid killing or restarting compactors while manual compactions are in flight","Bound the wait with a timeout instead of waiting on a dropped channel indefinitely"],"tags":["compaction","channel","compactor","timeout"],"backgroundTag":"request-timeout","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}