risingwavelabs/risingwave · error · anyhow::Error

reload iceberg table

Error message

reload iceberg table

What it means

The commit_retry helper `reload_table` reloads the table from the catalog and then asserts the current schema id equals the schema id the data files were written with; a mismatch means schema evolution happened externally and iceberg sink commits with the old schema are not allowed. The load itself or the schema check failing surfaces through this function in run_with_retry.

Source

Thrown at src/connector/src/sink/iceberg/commit_retry.rs:100

    /// `Transaction::commit` (or its `apply`) failed. Retriable — likely a
    /// commit conflict or transient network error.
    Commit(anyhow::Error),
}

/// Reload the iceberg table from the catalog and assert that its current
/// `schema_id` and `default_partition_spec_id` still match the values the
/// caller computed against. Schema or partition evolution mid-commit is
/// surfaced as a non-retriable error by the call sites.
pub async fn reload_table(
    catalog: &dyn Catalog,
    table_ident: &TableIdent,
    schema_id: i32,
    partition_spec_id: i32,
) -> Result<Table> {
    let table = catalog
        .load_table(table_ident)
        .await
        .map_err(|e| anyhow!(e).context("reload iceberg table"))?;
    if table.metadata().current_schema_id() != schema_id {
        bail!(
            "iceberg sink: schema evolution not supported; expect schema id {}, got {}",
            schema_id,
            table.metadata().current_schema_id(),
        );
    }
    if table.metadata().default_partition_spec_id() != partition_spec_id {
        bail!(
            "iceberg sink: partition evolution not supported; expect partition spec id {}, got {}",
            partition_spec_id,
            table.metadata().default_partition_spec_id(),
        );
    }
    Ok(table)
}

/// Run a commit-action against the given iceberg table with retry.

View on GitHub (pinned to 6469eb736d)

Solutions

1. Restart or recreate the RisingWave sink so it picks up the table's new current schema id.
2. Ensure schema changes flow only through RisingWave's sink schema-change path so ids stay in sync.
3. If load_table failed, fix catalog connectivity/credentials and retry.
4. Coordinate external writers to not evolve the schema while the sink is running.

Example fix

// before
if table.metadata().current_schema_id() != schema_id {
    bail!("iceberg sink: schema evolution not supported; expect schema id {}, got {}", schema_id, table.metadata().current_schema_id());
}
// after: reload and reconcile via the schema-change path instead of failing
if table.metadata().current_schema_id() != schema_id {
    return Err(SinkError::Iceberg(anyhow!(
        "schema id mismatch: sink expects {}, table is at {}; run commit_schema_change to reconcile",
        schema_id, table.metadata().current_schema_id()
    )));
}

// After any external DDL, verify the sink's expected schema id:
let table = catalog.load_table(&ident).await?;
if table.metadata().current_schema_id() != sink_schema_id {
    // reconcile: restart sink or run commit_schema_change
}

fn schema_in_sync(table: &Table, sink_schema_id: i32) -> bool {
    table.metadata().current_schema_id() == sink_schema_id
}

match run_with_retry(...).await {
    Err(e) if msg_contains(&e, "schema evolution not supported") => reconcile_sink_schema().await,
    Err(e) => alert(&e),
    Ok(_) => {}
}

When it happens

Trigger: Another writer (Spark/Flink/manual DDL) added/changed columns on the Iceberg table after the RisingWave sink cached its schema id, so `table.metadata().current_schema_id() != schema_id`; also triggered when load_table itself fails (catalog outage, table dropped).

Common situations: Concurrent schema evolution by an external engine; a RisingWave schema-change path updated the table but the sink retry still carries the old schema id; migration of catalogs yielding different schema ids.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• schema_id and partition_spec_id should be the same in all wr
• commit iceberg transaction
• Unsupported sink schema change op in iceberg sink: {:?}
• Current iceberg schema does not match either original_schema
• Failed to convert Arrow type to Iceberg type

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/3b86a37c07332c17. Report an issue: GitHub.