risingwavelabs/risingwave · error · anyhow::Error

iceberg sink: partition evolution not supported; expect part

Error message

iceberg sink: partition evolution not supported; expect partition spec id {}, got {}

What it means

Analogous to the schema check, the Iceberg sink compares the table's default partition spec id against the one recorded when the sink was created. If the partition spec changed externally (partition evolution, e.g. day->hour bucketing), the sink aborts the commit because this library does not support partition evolution. This prevents writing data files with an old partitioning layout into a table with a new spec.

Source

Thrown at src/connector/src/sink/iceberg/commit_retry.rs:109

pub async fn reload_table(
    catalog: &dyn Catalog,
    table_ident: &TableIdent,
    schema_id: i32,
    partition_spec_id: i32,
) -> Result<Table> {
    let table = catalog
        .load_table(table_ident)
        .await
        .map_err(|e| anyhow!(e).context("reload iceberg table"))?;
    if table.metadata().current_schema_id() != schema_id {
        bail!(
            "iceberg sink: schema evolution not supported; expect schema id {}, got {}",
            schema_id,
            table.metadata().current_schema_id(),
        );
    }
    if table.metadata().default_partition_spec_id() != partition_spec_id {
        bail!(
            "iceberg sink: partition evolution not supported; expect partition spec id {}, got {}",
            partition_spec_id,
            table.metadata().default_partition_spec_id(),
        );
    }
    Ok(table)
}

/// Run a commit-action against the given iceberg table with retry.
/// 1. Calls `reload_table` before each commit attempt to get the latest metadata
/// 2. If `reload_table` fails (table not exists/schema/partition mismatch), stops retrying immediately
/// 3. If commit fails, retries with backoff up to `retry_num` times.
///
/// Strategy: exponential backoff 10ms→60s with jitter, up to `retry_num` retries.
pub async fn run_with_retry<F, Fut, Out>(
    catalog: Arc<dyn Catalog>,
    table_ident: TableIdent,
    schema_id: i32,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Recreate the RisingWave iceberg sink so it binds to the table's current partition spec id
2. Revert the external partition-spec change so default_partition_spec_id matches the sink's expected id
3. Point the sink at a dedicated table not touched by other engines' partition evolution
4. Coordinate partition changes with sink downtime windows

Example fix

// before: partition spec changed externally
//   expect partition spec id 0, got 1
// after: recreate the sink
DROP SINK my_iceberg_sink;
CREATE SINK my_iceberg_sink AS ... WITH (
  connector = 'iceberg',
  table.name = 'db.my_table'  -- binds to current partition spec
);

let table = catalog.load_table(&table_ident).await?;
if table.metadata().default_partition_spec_id() != expected_partition_spec_id {
    return Err(anyhow!(
        "iceberg table partition spec changed (expect {}, got {}); recreate the sink",
        expected_partition_spec_id, table.metadata().default_partition_spec_id()
    ));
}

fn partition_spec_matches(table: &Table, expected_spec_id: i32) -> bool {
    table.metadata().default_partition_spec_id() == expected_spec_id
}

match run_with_retry(...).await {
    Err(e) if e.to_string().contains("partition evolution not supported") => {
        // partition spec changed externally: recreate the sink
        recreate_sink().await?;
    }
    Err(e) => return Err(e),
    Ok(_) => {}
}

When it happens

Trigger: `reload_table` during `run_with_retry` finds `table.metadata().default_partition_spec_id() != partition_spec_id`, i.e. the table's default partition spec was changed by an external tool between sink creation and this commit.

Common situations: Someone re-partitions the Iceberg table with Spark (REPLACE ... PARTITION) or replaces the table with a different partitioning; a table drop-and-recreate resets/changes the spec id; concurrent writers from different engines disagree on partitioning.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• iceberg sink: schema evolution not supported; expect schema
• Invalid order key item `{item}`: `NULLS` must be followed by
• Invalid order key item `{item}` HINT: `NULLS` must be follow
• Invalid order key token `{token}` in `{item}` HINT: Supporte
• order_key must not be empty

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/cf040a568354f923. Report an issue: GitHub.