risingwavelabs/risingwave · error

iceberg pk-index sink coordinator for sink {} timed out afte

Error message

iceberg pk-index sink coordinator for sink {} timed out after {}s loading iceberg catalog/table

What it means

IcebergPkIndexSinkCoordinator::init fails when load_catalog_and_table does not complete within the fixed 60s INIT_TIMEOUT. This means connecting to the Iceberg catalog and loading the target table took too long; the coordinator cannot be built so the sink cannot start serving commits.

Source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs:112

    retry_num: usize,
    /// The epoch pre-committed but not yet committed, carried from `pre_commit` to the next `commit`.
    waiting_commit: Option<EpochCommit>,
    prev_committed_epoch: Option<u64>,
}

impl IcebergPkIndexSinkCoordinator {
    /// Build a ready-to-serve coordinator: load the iceberg catalog/table, recover any persisted pending
    /// state, and drain recovered pending epochs to iceberg. Returns only once recovery is complete, so a
    /// successful return means the sink is ready to accept live pre-commit/commit calls.
    pub async fn init(
        sink_id: SinkId,
        iceberg_config: IcebergConfig,
        db: DatabaseConnection,
    ) -> Result<Self> {
        let (catalog, table) = timeout(INIT_TIMEOUT, load_catalog_and_table(&iceberg_config))
            .await
            .map_err(|_| {
                anyhow!(
                    "iceberg pk-index sink coordinator for sink {} timed out after {}s loading iceberg catalog/table",
                    sink_id,
                    INIT_TIMEOUT.as_secs()
                )
            })?
            .with_context(|| format!("init iceberg pk-index sink coordinator for sink {}", sink_id))?;

        let (prev_committed_epoch, recovered) =
            recovery(&db, sink_id).await.with_context(|| {
                format!(
                    "recover pending state for iceberg pk-index sink {}",
                    sink_id
                )
            })?;

        let target_branch =
            commit_branch(iceberg_config.r#type.as_str(), iceberg_config.write_mode);

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify iceberg catalog configuration (endpoint URL, warehouse, credentials) and test reachability from the meta node
2. Fix network/DNS or firewall issues blocking the catalog endpoint
3. Check catalog service health; retry sink creation after the catalog recovers
4. Reduce table metadata load (expire old snapshots) if the table is huge and the load legitimately exceeds 60s
5. Check the wrapped context 'init iceberg pk-index sink coordinator for sink {}' error for the underlying cause

Example fix

// before (config)
iceberg.endpoint = "http://internal-rest:8181" // unreachable -> 60s timeout
// after
iceberg.endpoint = "http://iceberg-rest.catalog.svc:8181" // verified reachable
// then: curl http://iceberg-rest.catalog.svc:8181/v1/config before creating the sink

// Before creating the sink, verify catalog reachability and config completeness
async fn validate_iceberg_catalog(cfg: &IcebergConfig) -> Result<()> {
    ensure!(!cfg.endpoint.is_empty(), "iceberg catalog endpoint is empty");
    let resp = reqwest::get(format!("{}/v1/config", cfg.endpoint)).await?;
    ensure!(resp.status().is_success(), "catalog endpoint not healthy");
    Ok(())
}

match IcebergPkIndexSinkCoordinator::init(sink_id, cfg.clone(), db.clone()).await {
    Err(e) if e.to_string().contains("timed out after 60s") => {
        warn!("catalog load timed out; checking endpoint then retrying");
        validate_iceberg_catalog(&cfg).await?;
        retry_with_backoff(|| init(sink_id, cfg.clone(), db.clone())).await
    }
    other => other,
}

When it happens

Trigger: Creating/starting an Iceberg pk-index sink: the coordinator's init calls timeout(60s, load_catalog_and_table(&iceberg_config)) and the elapsed timer fires before the catalog/table load returns.

Common situations: Wrong or unreachable catalog endpoint (REST/Hive/Glue) in iceberg config, missing or wrong credentials causing auth retries, DNS/network issues in the cluster, very large table metadata (tens of thousands of snapshots/files) making load slow, catalog service outage.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• create iceberg catalog for pk-index sink
• load iceberg table for pk-index sink
• `catalog.type` must be set
• Failed to list iceberg namespaces.
• Failed to load iceberg table.

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/ed6cef052677923f. Report an issue: GitHub.