apache/iceberg · warning

The configured equality field column IDs {} are not matched

Error message

The configured equality field column IDs {} are not matched with the schema identifier field IDs {}, use job specified equality field columns as the equality fields by default.

What it means

Warning from FlinkSink.checkAndGetEqualityFieldIds when the equality-field columns configured on the job do not exactly match the table schema's identifier field IDs. The sink proceeds using the job-specified columns as equality fields, which can differ from what upsert semantics the table's identifier fields imply.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java:512

    @VisibleForTesting
    List<Integer> checkAndGetEqualityFieldIds() {
      List<Integer> equalityFieldIds = Lists.newArrayList(table.schema().identifierFieldIds());
      if (equalityFieldColumns != null && !equalityFieldColumns.isEmpty()) {
        Set<Integer> equalityFieldSet =
            Sets.newHashSetWithExpectedSize(equalityFieldColumns.size());
        for (String column : equalityFieldColumns) {
          org.apache.iceberg.types.Types.NestedField field = table.schema().findField(column);
          Preconditions.checkNotNull(
              field,
              "Missing required equality field column '%s' in table schema %s",
              column,
              table.schema());
          equalityFieldSet.add(field.fieldId());
        }

        if (!equalityFieldSet.equals(table.schema().identifierFieldIds())) {
          LOG.warn(
              "The configured equality field column IDs {} are not matched with the schema identifier field IDs"
                  + " {}, use job specified equality field columns as the equality fields by default.",
              equalityFieldSet,
              table.schema().identifierFieldIds());
        }
        equalityFieldIds = Lists.newArrayList(equalityFieldSet);
      }
      return equalityFieldIds;
    }

    private DataStreamSink<Void> appendDummySink(SingleOutputStreamOperator<Void> committerStream) {
      DataStreamSink<Void> resultStream =
          committerStream
              .sinkTo(new DiscardingSink<>())
              .name(operatorName(String.format("IcebergSink %s", this.table.name())))
              .setParallelism(1);
      if (uidPrefix != null) {
        resultStream = resultStream.uid(uidPrefix + "-dummysink");

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Align setEqualityFieldColumns(...) with the table's identifier fields, or remove it so the schema identifier fields are used
2. Verify current identifier field IDs with DESCRIBE TABLE / table.schema().identifierFieldIds()
3. If the job-specified columns are intentionally different, silence the warning by acknowledging the divergence in job docs/config review
4. Update and redeploy the Flink job after table identifier-field changes

Example fix

// before
FlinkSink.forRowData(input)
    .setEqualityFieldColumns("order_id")
    ...
// after (match schema identifier fields, e.g. [order_id, line_number])
FlinkSink.forRowData(input)
    .setEqualityFieldColumns("order_id", "line_number")
    ...

Set<Integer> configured = equalityFieldSet;
Set<Integer> identifiers = table.schema().identifierFieldIds();
if (!identifiers.isEmpty() && !configured.equals(identifiers)) {
  throw new IllegalArgumentException("equality columns must match identifier fields: " + identifiers);
}

When it happens

Trigger: Writing with FlinkSink in upsert mode while setEqualityFieldColumns picks columns whose resolved field IDs differ from table.schema().identifierFieldIds() — e.g. different columns, a subset, or the table's identifier fields changed after the job was configured.

Common situations: Table schema identifier fields updated (ALTER TABLE ... SET IDENTIFIER FIELDS) while the Flink job still passes old equality columns; typo'd or reordered column names; job config and table evolution drift.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• The configured equality field column IDs {} are not matched
• The configured equality field column IDs {} are not matched
• The configured equality field column IDs {} are not matched
• Invalid primary key '%s'. A primary key must not contain dup
• Invalid primary key '%s'. Column '%s' does not exist.

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/40141e20fd006d5b. Report an issue: GitHub.