risingwavelabs/risingwave · error

commit iceberg pk-index sink transaction

Error message

commit iceberg pk-index sink transaction

What it means

After the overwrite action is applied to the transaction, `txn.commit(catalog)` performs the CAS commit against the catalog. Failures (commit conflict with a concurrent writer, catalog HTTP error, timeout) are wrapped with this context as `CommitError::Commit`.

Source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs:537

                    .iter()
                    .map(&materialize)
                    .collect::<Result<Vec<_>, _>>()?;

                let txn = Transaction::new(&table);
                let action = txn
                    .overwrite_files()
                    .set_snapshot_id(snapshot_id)
                    .set_target_branch(target_branch)
                    .add_data_files(add_files)
                    .delete_files(overwrite_files);
                let txn = action.apply(txn).map_err(|err| {
                    CommitError::Commit(
                        anyhow!(err).context("apply iceberg pk-index sink overwrite_files action"),
                    )
                })?;
                let table = txn.commit(catalog.as_ref()).await.map_err(|err| {
                    CommitError::Commit(
                        anyhow!(err).context("commit iceberg pk-index sink transaction"),
                    )
                })?;
                Ok(table)
            }
        },
    )
    .await
    .map_err(CommitError::Commit)
}

#[derive(Clone, Serialize, Deserialize)]
struct IcebergPkIndexSinkAggResult {
    schema_id: i32,
    partition_spec_id: i32,
    data_files: Vec<SerializedDataFile>,
    delete_files: Vec<SerializedDataFile>,
    overwrite_files: Vec<SerializedDataFile>,
}

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check the inner error: if it's a requirements/commit conflict, ensure the built-in commit retry (run_with_retry) exhausted its attempts and consider raising retry count.
2. Pause/avoid external writers (compaction jobs) on the sink-managed table, or schedule them outside sink commit windows.
3. Verify catalog availability and network from the meta node; increase catalog request timeout if commits are large.

Example fix

null

// preflight catalog reachability
// curl -sSf -X POST $CATALOG_URI/v1/{prefix}/tables/commit -o /dev/null --max-time 10

match txn.commit(catalog.as_ref()).await {
    Ok(table) => Ok(table),
    Err(e) if is_conflict(&e) => {
        // retryable: reload latest table and re-run run_with_retry
        Err(CommitError::Commit(anyhow!(e).context("commit iceberg pk-index sink transaction")))
    }
    Err(e) => Err(CommitError::Commit(anyhow!(e).context("commit iceberg pk-index sink transaction"))),
}

When it happens

Trigger: `txn.commit(catalog.as_ref())` fails in `commit_one_epoch`: another engine committed to the table concurrently causing a requirements conflict, the catalog is unreachable, or the commit request times out / returns an error.

Common situations: Concurrent compaction or Spark jobs committing to the same Iceberg table; REST catalog rate limiting or downtime; long-running commits exceeding catalog request timeouts; repeated conflicts exhausting `retry_num`.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• commit iceberg transaction
• `catalog.type` must be set
• Failed to list iceberg namespaces.
• Failed to load iceberg table.
• Failed to drop iceberg table.

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d27241e8fa93ad3a. Report an issue: GitHub.