{"record":{"id":"99318374a83e6d19","repo":"risingwavelabs/risingwave","slug":"manual-iceberg-compaction-waiter-dropped-unexpecte","errorCode":null,"errorMessage":"Manual iceberg compaction waiter dropped unexpectedly for sink {}","messagePattern":"Manual iceberg compaction waiter dropped unexpectedly for sink (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/manager/iceberg_compaction/manual.rs","lineNumber":79,"sourceCode":"\n        let waiter = self.start_manual_compaction(sink_id).await?;\n        let cleanup_guard = scopeguard::guard(sink_id, |sink_id| {\n            self.cancel_manual_compaction_waiter(sink_id)\n        });\n\n        tracing::info!(\n            \"Manual compaction requested for sink {}, waiting for completion...\",\n            sink_id\n        );\n\n        match waiter.await {\n            Ok(result) => {\n                let _ = scopeguard::ScopeGuard::into_inner(cleanup_guard);\n                result\n            }\n            Err(_) => {\n                let _ = scopeguard::ScopeGuard::into_inner(cleanup_guard);\n                Err(anyhow!(\n                    \"Manual iceberg compaction waiter dropped unexpectedly for sink {}\",\n                    sink_id,\n                )\n                .into())\n            }\n        }\n    }\n}\n","sourceCodeStart":61,"sourceCodeEnd":88,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/manager/iceberg_compaction/manual.rs#L61-L88","documentation":"trigger_manual_compaction waits on a one-shot channel registered by start_manual_compaction. If the channel's sender side is dropped before delivering a result (the scheduler dropped the waiter without sending), recv() returns Err and the meta reports the waiter as dropped unexpectedly, since this indicates an internal scheduling bug rather than a compaction failure.","triggerScenarios":"The recv on the manual-compaction waiter returns a RecvError, meaning the schedule-side sender was dropped (e.g. scheduler entry removed without completing the waiter).","commonSituations":"Internal race where the task is cancelled/cleaned without calling the waiter's fail path (see the schedule.rs Drop handler); meta node shutdown mid-wait; bugs in manual task bookkeeping.","solutions":["Retry the manual compaction; this is usually a transient internal cleanup race.","Check meta logs for concurrent cancellation of the sink's compaction task.","If reproducible, inspect IcebergCompactionScheduler Drop/cancel paths for a missing waiter.send(Err(...)).","File a bug with sink_id and meta logs; sender-drop without result is an invariant violation."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match trigger_manual_compaction(sink_id).await {\n    Err(e) if e.to_string().contains(\"waiter dropped unexpectedly\") => {\n        // transient scheduler race: retry once with backoff\n        tokio::time::sleep(Duration::from_secs(2)).await;\n        trigger_manual_compaction(sink_id).await\n    }\n    other => other,\n}","preventionTips":["Avoid shutting down meta while manual compactions are pending.","Ensure every scheduler cancellation path completes its waiter (send Err before drop).","Report persistent occurrences as scheduler bookkeeping bugs."],"tags":["rust","meta-service","iceberg","compaction","channel"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}