{"record":{"id":"d4536f41b936f3fa","repo":"risingwavelabs/risingwave","slug":"iceberg-compaction-task-failed-before-dispatch-for","errorCode":null,"errorMessage":"Iceberg compaction task failed before dispatch for sink {}","messagePattern":"Iceberg compaction task failed before dispatch for sink (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/manager/iceberg_compaction/schedule.rs","lineNumber":572,"sourceCode":"                Some(track.revert_pre_dispatch_failure(Instant::now()))\n            } else {\n                None\n            };\n            if let Some(finish_action) = finish_action {\n                let waiter = guard.manual_compaction_waiters.remove(&self.sink_id);\n                IcebergCompactionManager::apply_track_finish_action(\n                    &mut guard,\n                    self.sink_id,\n                    finish_action,\n                );\n                waiter\n            } else {\n                None\n            }\n        };\n\n        if let Some(waiter) = waiter {\n            let _ = waiter.send(Err(anyhow!(\n                \"Iceberg compaction task failed before dispatch for sink {}\",\n                self.sink_id\n            )\n            .into()));\n        }\n    }\n}\n\n#[derive(Debug, Clone)]\nenum SinkUpdateKind {\n    /// A normal sink commit. It increases the pending snapshot count.\n    Commit {\n        observed_snapshot: IcebergCommittedSnapshot,\n    },\n    /// A force signal from the sink update path. It triggers the configured\n    /// compaction type and still follows the automatic-compaction config gate.\n    ForceCompaction {\n        observed_snapshot: IcebergCommittedSnapshot,","sourceCodeStart":554,"sourceCodeEnd":590,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/manager/iceberg_compaction/schedule.rs#L554-L590","documentation":"The scheduler entry's Drop implementation must never leak a registered manual-compaction waiter: if the task is dropped before being dispatched to a compactor, it sends a pre-formatted error through the waiter channel so trigger_manual_compaction fails fast instead of hanging forever.","triggerScenarios":"An IcebergCompactionScheduler task struct is dropped before dispatch (e.g. queue eviction, cancellation, scheduler shutdown, early error return) while a waiter Option is still Some.","commonSituations":"Meta node shutting down with pending manual compaction tasks; task cancelled/evicted before a compactor picked it up; internal bugs dropping the scheduler entry without completing the waiter.","solutions":["Retry the manual compaction after the scheduler is healthy.","Check meta logs for scheduler shutdown/eviction around the failure time.","If this happens without shutdown/cancellation, it indicates a Drop-path bug; report with sink_id and logs.","Ensure compactor workers are online so tasks get dispatched instead of evicted."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match trigger_manual_compaction(sink_id).await {\n    Err(e) if e.to_string().contains(\"failed before dispatch\") => {\n        // scheduler dropped the task early; retry while scheduler is healthy\n        retry_with_backoff(3, || trigger_manual_compaction(sink_id)).await\n    }\n    other => other,\n}","preventionTips":["Avoid meta restarts while compaction tasks are queued.","Keep compactor workers online so tasks dispatch promptly.","Monitor for tasks failing before dispatch — persistent cases indicate a scheduler bug."],"tags":["rust","meta-service","iceberg","compaction","task-scheduling"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}