risingwavelabs/risingwave · error · SinkError

iceberg catalog did not reflect committed pk-index snapshot

Error message

iceberg catalog did not reflect committed pk-index snapshot {:?} after {} attempts (last current_snapshot_id={:?})

What it means

load_table_at_least polls the Iceberg catalog until its loaded table metadata includes the expected committed snapshot id (the pk-index sink's committed snapshot) and gives up after MAX_ATTEMPTS with this error. It exists because a commit by the sink may not be immediately visible to the merger's catalog load (eventual consistency / propagation lag). If the catalog still shows an older current_snapshot_id after all retries, seeding the merger's position-delete staging is aborted.

Source

Thrown at src/stream/src/executor/iceberg_with_pk_index/mod.rs:65

pub async fn load_table_at_least(
    config: &IcebergConfig,
    expected: Option<i64>,
) -> SinkResult<Table> {
    const MAX_ATTEMPTS: usize = 10;
    const BACKOFF: Duration = Duration::from_millis(500);
    let mut last = None;
    for _ in 0..MAX_ATTEMPTS {
        let table = config.load_table().await?;
        let Some(expected) = expected else {
            return Ok(table);
        };
        if table.metadata().snapshot_by_id(expected).is_some() {
            return Ok(table);
        }
        last = Some(table.metadata().current_snapshot_id());
        tokio::time::sleep(BACKOFF).await;
    }
    Err(SinkError::Iceberg(anyhow::anyhow!(
        "iceberg catalog did not reflect committed pk-index snapshot {:?} after {} attempts (last current_snapshot_id={:?})",
        expected,
        MAX_ATTEMPTS,
        last,
    )))
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the pk-index sink actually committed the expected snapshot (check sink metrics/logs and the catalog's table history).
2. Check catalog health and reduce metadata staleness (disable/shorten catalog cache TTL, ensure REST catalog doesn't serve stale metadata).
3. Inspect network connectivity between the merger node and the catalog service.
4. Increase MAX_ATTEMPTS or BACKOFF if the catalog is known to be slow under load.
5. Retry the job — a transient catalog hiccup may resolve on re-seed.

Example fix

// before
Err(SinkError::Iceberg(anyhow::anyhow!(
    "iceberg catalog did not reflect committed pk-index snapshot {:?} after {} attempts ..."
)))
// after
// No code fix; retry the sink/merger after confirming the catalog is reachable
// and that the sink's commit succeeded, e.g.:
// SELECT * FROM "iceberg_catalog_history" WHERE table = '...';  -- verify snapshot exists

// Rust (operator-side check before relying on merger seed)
let table = catalog.load_table(&ident).await?;
let expected = /* committed snapshot id from sink */;
if table.metadata().snapshot_by_id(expected).is_none() {
    eprintln!("catalog lagging: current={:?}, expected={:?}",
        table.metadata().current_snapshot_id(), expected);
}

fn catalog_has_snapshot(table: &Table, expected: i64) -> bool {
    table.metadata().snapshot_by_id(expected).is_some()
}

// wrap seeding with retry-and-diagnose
match start_seed(...).await {
    Err(SinkError::Iceberg(e)) if format!("{:e}", e).contains("did not reflect committed pk-index snapshot") => {
        warn!("catalog lagging or commit missing; verify sink commit then retry");
    }
    other => other?,
}

When it happens

Trigger: Raised by load_table_at_least (called from resolve and start_seed) when, after MAX_ATTEMPTS backoff sleeps, table.metadata().snapshot_by_id(expected) is still None and current_snapshot_id remains below/behind the expected snapshot id returned by wait_iceberg_pk_index_sink_epoch.

Common situations: Catalog backends with cached/laggy metadata (e.g. REST catalogs with stale cache, or a catalog whose refresh interval exceeds the retry window); the pk-index sink commit failed silently so the expected snapshot never exists; network issues between merger and catalog; clocks/attempts too tight for a heavily loaded catalog.

Related errors

• `catalog.type` must be set
• Failed to list iceberg namespaces.
• Failed to load iceberg table.
• Failed to drop iceberg table.
• register_table is not supported by JniCatalog

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/412a988586544509. Report an issue: GitHub.