risingwavelabs/risingwave · warning

iceberg_compaction_send_task_fail

Error message

iceberg_compaction_send_task_fail

What it means

This is a fail-point (fault-injection) error inside `IcebergCompactorManager::send_event` (src/meta/src/hummock/compactor_manager.rs:504). `fail_point!("iceberg_compaction_send_task_fail")` makes the meta node return an artificial error whenever the fail point is activated, simulating a failure to send a subscribe-response event to the iceberg compaction task. It is a testing hook: in production, send_event normally just pushes the event into a channel and cannot fail with this message.

Source

Thrown at src/meta/src/hummock/compactor_manager.rs:504

        self.inner.read().get_progress()
    }
}

impl IcebergCompactor {
    pub fn new(
        context_id: HummockContextId,
        sender: IcebergCompactorSubscribeStreamSender,
    ) -> Self {
        Self { context_id, sender }
    }

    pub fn context_id(&self) -> HummockContextId {
        self.context_id
    }

    pub fn send_event(&self, event: IcebergCompactorSubscribeResponseEvent) -> MetaResult<()> {
        fail_point!("iceberg_compaction_send_task_fail", |_| Err(
            anyhow::anyhow!("iceberg_compaction_send_task_fail").into()
        ));

        self.sender
            .send(Ok(SubscribeIcebergCompactionEventResponse {
                create_at: SystemTime::now()
                    .duration_since(SystemTime::UNIX_EPOCH)
                    .expect("Clock may have gone backwards")
                    .as_millis() as u64,
                event: Some(event),
            }))
            .map_err(|e| anyhow::anyhow!(e))?;

        Ok(())
    }

    pub fn cancel_task(&self, task_id: IcebergCompactionTaskId) -> MetaResult<()> {
        self.send_event(IcebergResponseEvent::CancelCompactTask(
            IcebergCancelCompactTask { task_id },

View on GitHub (pinned to 6469eb736d)

Solutions

1. Disable the `iceberg_compaction_send_task_fail` fail point in the runtime configuration/environment
2. If it appears unintentionally, check FAIL_POINTS / fail-point configuration inherited from test runs
3. Retry the iceberg compaction task; the error is synthetic, not a real channel failure
4. If testing, use the fail point's configured closure behavior to control recovery

Example fix

// before (env with fail point armed)
FAIL_POINTS=iceberg_compaction_send_task_fail ./risedev d
// after
unset FAIL_POINTS  # or remove the fail point entry before running

// Only run with fail points armed in test environments
assert!(cfg!(debug_assertions) || env::var("FAIL_POINTS").is_err(),
    "fail points must not be armed in production");

fn is_fail_point_error(e: &MetaError) -> bool {
    e.to_string().contains("iceberg_compaction_send_task_fail")
}

match send_event_result {
    Err(e) if e.to_string().contains("fail") => {
        tracing::warn!("fail-point triggered send_event error; retrying");
        retry(send_event)
    }
    other => other,
}

When it happens

Trigger: The fail point is enabled (via the FAIL_POINTS env/config mechanism, e.g. in tests or `risedev` fault-injection runs) while an iceberg compactor reports an event through `send_event`, called e.g. from `cancel_task` or the compactor subscribe-response handler.

Common situations: Only occurs during chaos/fault-injection testing or when a developer has explicitly armed this fail point; seeing it in a normal deployment means a test configuration leaked into production.

Related errors

• `compaction_interval_sec` must be greater than 0
• `compaction.trigger_snapshot_count` must be greater than 0
• `compaction.max_snapshots_num` must be greater than 0
• `compaction.small_files_threshold_mb` must be greater than 0
• `compaction.delete_files_count_threshold` must be greater th

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/a8b5113d1627b87f. Report an issue: GitHub.