apache/iceberg · warning

Hash distribute rows by equality fields, even though {}=rang

Error message

Hash distribute rows by equality fields, even though {}=range is set. Range distribution for primary keys are not always safe in Flink streaming writer.

What it means

Same family as the FlinkSink range warning: when WRITE_DISTRIBUTION_MODE is 'range' but equality fields (primary key / upsert) are present, IcebergSink cannot apply range distribution safely for streaming writes and instead hashes rows by the equality fields via keyBy, logging this warning for backward compatibility rather than failing.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java:1050

    return Optional.ofNullable(flinkWriteConf.writeParallelism()).orElseGet(input::getParallelism);
  }

  private DataStream<RowData> distributeDataStreamByRangeDistributionMode(
      DataStream<RowData> input,
      Schema iSchema,
      PartitionSpec partitionSpec,
      SortOrder sortOrderParam) {

    int writerParallelism = resolveWriterParallelism(input);

    // needed because of checkStyle not allowing us to change the value of an argument
    SortOrder sortOrder = sortOrderParam;

    // Ideally, exception should be thrown in the combination of range distribution and
    // equality fields. Primary key case should use hash distribution mode.
    // Keep the current behavior of falling back to keyBy for backward compatibility.
    if (!equalityFieldIds.isEmpty()) {
      LOG.warn(
          "Hash distribute rows by equality fields, even though {}=range is set. "
              + "Range distribution for primary keys are not always safe in "
              + "Flink streaming writer.",
          WRITE_DISTRIBUTION_MODE);
      return input.keyBy(new EqualityFieldKeySelector(iSchema, flinkRowType, equalityFieldIds));
    }

    // range distribute by partition key or sort key if table has an SortOrder
    Preconditions.checkState(
        sortOrder.isSorted() || partitionSpec.isPartitioned(),
        "Invalid write distribution mode: range. Need to define sort order or partition spec.");
    if (sortOrder.isUnsorted()) {
      sortOrder = Partitioning.sortOrderFor(partitionSpec);
      LOG.info("Construct sort order from partition spec");
    }

    LOG.info("Range distribute rows by sort order: {}", sortOrder);
    StatisticsOrRecordTypeInformation statisticsOrRecordTypeInformation =

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Switch to distributionMode(DistributionMode.HASH) for primary-key streaming writes
2. If sorting is the goal, use batch mode or ensure your distribution/sort intent does not rely on range mode in streaming
3. Verify the equality fields are correct since they now determine record placement

Example fix

// before
IcebergSink.builder().upsert(true).distributionMode(DistributionMode.RANGE).append();
// after
IcebergSink.builder().upsert(true).distributionMode(DistributionMode.HASH).append();

if (DistributionMode.RANGE.equals(distributionMode) && !equalityFieldIds.isEmpty()) {
  throw new IllegalArgumentException("Use hash distribution with equality fields");
}

When it happens

Trigger: IcebergSink built with .equalityFields(...) (upsert) while the table or builder sets write.distribution-mode=range (DistributionMode.RANGE), possibly with a sortOrder configured.

Common situations: Tables configured with range distribution + sort order reused in Flink upsert streaming jobs; configs copied from batch engines; users expecting sorted output files who instead get hash-by-key distribution.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Hash distribute rows by equality fields, even though {}=rang
• ${tableName}: Unrecognized ${WRITE_DISTRIBUTION_MODE}: ${mod
• Unrecognized :
• Hash distribute rows by equality fields, even though {}=rang
• The configured equality field column IDs {} are not matched

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/264e6ec8e4e91826. Report an issue: GitHub.