The configured equality field column IDs {} are not matched

Error message

The configured equality field column IDs {} are not matched with the schema identifier field IDs {}, use job specified equality field columns as the equality fields by default.

What it means

SinkUtil.checkAndGetEqualityFieldIds warns when the equality-field columns configured for the job resolve to field IDs that differ from the table schema's identifierFieldIds (primary keys). The job-specified columns are used as equality fields by default; the mismatch typically means the table's identifier fields evolved after the job config was written.

Source

Thrown at flink/v2.2/flink/src/main/java/org/apache/iceberg/flink/sink/SinkUtil.java:74

  private static final Logger LOG = LoggerFactory.getLogger(SinkUtil.class);

  static Set<Integer> checkAndGetEqualityFieldIds(Table table, List<String> equalityFieldColumns) {
    Set<Integer> equalityFieldIds = Sets.newHashSet(table.schema().identifierFieldIds());
    if (equalityFieldColumns != null && !equalityFieldColumns.isEmpty()) {
      Set<Integer> equalityFieldSet = Sets.newHashSetWithExpectedSize(equalityFieldColumns.size());
      for (String column : equalityFieldColumns) {
        org.apache.iceberg.types.Types.NestedField field = table.schema().findField(column);
        Preconditions.checkNotNull(
            field,
            "Missing required equality field column '%s' in table schema %s",
            column,
            table.schema());
        equalityFieldSet.add(field.fieldId());
      }

      if (!equalityFieldSet.equals(table.schema().identifierFieldIds())) {
        LOG.warn(
            "The configured equality field column IDs {} are not matched with the schema identifier field IDs"
                + " {}, use job specified equality field columns as the equality fields by default.",
            equalityFieldSet,
            table.schema().identifierFieldIds());
      }
      equalityFieldIds = Sets.newHashSet(equalityFieldSet);
    }
    return equalityFieldIds;
  }

  static long getMaxCommittedCheckpointId(
      Table table, String flinkJobId, String operatorId, String branch) {
    Snapshot snapshot = table.snapshot(branch);
    long lastCommittedCheckpointId = INITIAL_CHECKPOINT_ID;

    while (snapshot != null) {
      Map<String, String> summary = snapshot.summary();
      String snapshotFlinkJobId = summary.get(FLINK_JOB_ID);

View on GitHub (pinned to 86d9c8fc54)

Solutions

Update equalityFieldColumns to match the table's identifier fields, or drop the explicit setting.
Inspect the current schema's identifierFieldIds to confirm the intended key.
Restart the job with corrected config so equality semantics match the table.

Example fix

// before
.equalityFieldColumns("order_id") // identifier fields now (order_id, region)
// after
.equalityFieldColumns("order_id", "region")

Defensive patterns

Strategy: validation

Validate before calling

Set<Integer> ids = SinkUtil.checkAndGetEqualityFieldIds(table, eqColumns);
if (!ids.equals(table.schema().identifierFieldIds())) {
  throw new IllegalArgumentException("configured equality fields differ from identifierFieldIds");
}

Prevention

Keep equalityFieldColumns in sync with schema identifier fields
Re-check after schema evolution or table migration
Prefer deriving equality fields from the schema

When it happens

Trigger: Calling SinkUtil.checkAndGetEqualityFieldIds (used by sink builders) with equalityFieldColumns set whose resolved ID set does not equal table.schema().identifierFieldIds().

Common situations: Primary keys changed via schema evolution after the job was configured; column-name typos/case differences resolving to different IDs; reusing sink configuration across tables.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d01f0a796870e751. Report an issue: GitHub.