risingwavelabs/risingwave · error

apply iceberg pk-index sink overwrite_files action

Error message

apply iceberg pk-index sink overwrite_files action

What it means

The coordinator builds an Iceberg `overwrite_files` transaction action (adding data/delete files and deleting overwritten files on a target branch) and applies it to the transaction. If the action's in-memory application fails (invalid requirement evaluation, empty/invalid operations), this error is thrown as `CommitError::Commit`.

Source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs:532

                        .map(&materialize)
                        .collect::<Result<Vec<_>, _>>()?,
                };
                let overwrite_files: Vec<DataFile> = merged
                    .overwrite_files
                    .iter()
                    .map(&materialize)
                    .collect::<Result<Vec<_>, _>>()?;

                let txn = Transaction::new(&table);
                let action = txn
                    .overwrite_files()
                    .set_snapshot_id(snapshot_id)
                    .set_target_branch(target_branch)
                    .add_data_files(add_files)
                    .delete_files(overwrite_files);
                let txn = action.apply(txn).map_err(|err| {
                    CommitError::Commit(
                        anyhow!(err).context("apply iceberg pk-index sink overwrite_files action"),
                    )
                })?;
                let table = txn.commit(catalog.as_ref()).await.map_err(|err| {
                    CommitError::Commit(
                        anyhow!(err).context("commit iceberg pk-index sink transaction"),
                    )
                })?;
                Ok(table)
            }
        },
    )
    .await
    .map_err(CommitError::Commit)
}

#[derive(Clone, Serialize, Deserialize)]
struct IcebergPkIndexSinkAggResult {
    schema_id: i32,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Read the inner error from the log to see which requirement/operation failed the apply step.
2. Confirm the target branch exists and is correct in the sink config ('target branch' option).
3. Rely on/verify the commit retry loop: if the failure is from concurrent commits, ensure `run_with_retry` reloads the latest table before reapplying.

Example fix

null

// ensure branch exists before applying
let branches: Vec<_> = table.metadata().branches().unwrap_or_default().collect();
anyhow::ensure!(branches.iter().any(|b| b == &target_branch) || target_branch == "main", "target branch {target_branch} missing");

let txn = match action.apply(txn) {
    Ok(t) => t,
    Err(e) => return Err(CommitError::Commit(anyhow!(e).context("apply iceberg pk-index sink overwrite_files action"))),
};

When it happens

Trigger: `action.apply(txn)` fails in `commit_one_epoch`: e.g. invalid overwrite operation given the table's requirements, snapshot/branch mismatch, or an iceberg-rust validation error when applying the update against the current table metadata.

Common situations: Target branch doesn't exist or was repointed; concurrent commits changed table state between table load and apply; iceberg-rust library version with stricter overwrite validation.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Failed to update iceberg table.
• Invalid commit metadata: empty payload
• Invalid commit metadata: missing snapshot_id
• apply iceberg fast_append
• commit iceberg transaction

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/4fea5b74ab48362d. Report an issue: GitHub.