risingwavelabs/risingwave · error

Manual iceberg compaction waiter dropped unexpectedly for si

Error message

Manual iceberg compaction waiter dropped unexpectedly for sink {}

What it means

trigger_manual_compaction waits on a one-shot channel registered by start_manual_compaction. If the channel's sender side is dropped before delivering a result (the scheduler dropped the waiter without sending), recv() returns Err and the meta reports the waiter as dropped unexpectedly, since this indicates an internal scheduling bug rather than a compaction failure.

Source

Thrown at src/meta/src/manager/iceberg_compaction/manual.rs:79

        let waiter = self.start_manual_compaction(sink_id).await?;
        let cleanup_guard = scopeguard::guard(sink_id, |sink_id| {
            self.cancel_manual_compaction_waiter(sink_id)
        });

        tracing::info!(
            "Manual compaction requested for sink {}, waiting for completion...",
            sink_id
        );

        match waiter.await {
            Ok(result) => {
                let _ = scopeguard::ScopeGuard::into_inner(cleanup_guard);
                result
            }
            Err(_) => {
                let _ = scopeguard::ScopeGuard::into_inner(cleanup_guard);
                Err(anyhow!(
                    "Manual iceberg compaction waiter dropped unexpectedly for sink {}",
                    sink_id,
                )
                .into())
            }
        }
    }
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Retry the manual compaction; this is usually a transient internal cleanup race.
2. Check meta logs for concurrent cancellation of the sink's compaction task.
3. If reproducible, inspect IcebergCompactionScheduler Drop/cancel paths for a missing waiter.send(Err(...)).
4. File a bug with sink_id and meta logs; sender-drop without result is an invariant violation.

match trigger_manual_compaction(sink_id).await {
    Err(e) if e.to_string().contains("waiter dropped unexpectedly") => {
        // transient scheduler race: retry once with backoff
        tokio::time::sleep(Duration::from_secs(2)).await;
        trigger_manual_compaction(sink_id).await
    }
    other => other,
}

When it happens

Trigger: The recv on the manual-compaction waiter returns a RecvError, meaning the schedule-side sender was dropped (e.g. scheduler entry removed without completing the waiter).

Common situations: Internal race where the task is cancelled/cleaned without calling the waiter's fail path (see the schedule.rs Drop handler); meta node shutdown mid-wait; bugs in manual task bookkeeping.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Manual iceberg compaction task {} for sink {} failed: {}
• No iceberg compactor available
• Iceberg compaction task failed before dispatch for sink {}
• manual iceberg compaction is already waiting for sink {}
• Iceberg branch {} disappeared after manifest rewrite for sin

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/99318374a83e6d19. Report an issue: GitHub.