{"record":{"id":"d6ba4ae3fb13d910","repo":"risingwavelabs/risingwave","slug":"iceberg-compaction-task-report-timed-out-for-sink","errorCode":null,"errorMessage":"Iceberg compaction task report timed out for sink {}","messagePattern":"Iceberg compaction task report timed out for sink (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/manager/iceberg_compaction/schedule.rs","lineNumber":1048,"sourceCode":"                .take(n)\n                .filter_map(|(sink_id, _)| {\n                    let track = guard.sink_schedules.get_mut(&sink_id)?;\n                    let attempt = track.start_attempt();\n\n                    Some(IcebergCompactionHandle::new(\n                        sink_id,\n                        attempt,\n                        self.inner.clone(),\n                        self.metadata_manager.clone(),\n                    ))\n                })\n                .collect();\n\n            (handles, timed_out_waiters)\n        };\n\n        for (sink_id, waiter) in timed_out_waiters {\n            let _ = waiter.send(Err(anyhow!(\n                \"Iceberg compaction task report timed out for sink {}\",\n                sink_id\n            )\n            .into()));\n        }\n\n        handles\n    }\n\n    pub fn clear_iceberg_maintenance_by_sink_id(&self, sink_id: SinkId) {\n        let (task_to_cancel, waiter) = {\n            let mut guard = self.inner.write();\n            let task_to_cancel = Self::remove_sink_schedule(&mut guard, sink_id);\n            guard.snapshot_expiration_sink_ids.remove(&sink_id);\n            guard.manifest_rewrite_sink_ids.remove(&sink_id);\n            let waiter = guard.manual_compaction_waiters.remove(&sink_id);\n            (task_to_cancel, waiter)\n        };","sourceCodeStart":1030,"sourceCodeEnd":1066,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/manager/iceberg_compaction/schedule.rs#L1030-L1066","documentation":"A manual Iceberg compaction waiter received an Err because the dispatched compaction task did not report within its report_deadline. The scheduler's finish_timed_out_compaction_tasks marks the task failed and resolves any pending manual-compaction oneshot waiter with this timeout error.","triggerScenarios":"A manual compaction was triggered (caller holds the oneshot::Receiver from start_manual_compaction); the in-flight task misses its report deadline, so finish_timed_out_compaction_tasks sends this error through the waiter channel.","commonSituations":"Worker executing the compaction task is overloaded, partitioned from meta, or crashed without reporting; task legitimately takes longer than the configured report timeout; long GC pause on the worker.","solutions":["Check worker/meta logs around the timeout for the sink's task to find why no report arrived","Retry the manual compaction after the track resets to Idle (the timed-out task is already marked failed)","Increase the report timeout if tasks legitimately run longer than the deadline","Fix worker health: connectivity to meta, resource saturation, or crashes"],"exampleFix":"// before\nlet result = rx.await??; // Err: report timed out\n// after\nmatch rx.await {\n    Ok(Ok(task_id)) => Ok(task_id),\n    Ok(Err(e)) if e.to_string().contains(\"report timed out\") => {\n        warn!(\"compaction report timed out; will retry after track resets\");\n        retry_manual_compaction(sink_id).await\n    }\n    r => r,\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"let result = rx.await;\nmatch result {\n    Ok(Ok(task_id)) => Ok(task_id),\n    Ok(Err(e)) if e.to_string().contains(\"report timed out\") => {\n        warn!(sink_id, \"manual compaction timed out; scheduler will reset track\");\n        retry_after_reset(sink_id).await\n    }\n    Ok(Err(e)) => Err(e),\n    Err(_recv_err) => Err(anyhow!(\"waiter dropped by scheduler\")),\n}","preventionTips":["Keep the oneshot receiver alive until resolution; dropping it cancels the waiter","Size report timeouts to accommodate worst-case compaction durations","Monitor for repeated report timeouts, which indicate worker health or network issues","Retry after the timeout because the scheduler automatically marks the task failed"],"tags":["iceberg","compaction","timeout","worker"],"backgroundTag":"request-timeout","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}