apache/iceberg · error · IllegalStateException

Internal algorithm error: exhausted subtasks with unassigned

Error message

Internal algorithm error: exhausted subtasks with unassigned keys left

What it means

MapAssignment.buildAssignment() throws IllegalStateException when the range-assignment algorithm runs out of subtasks while map keys (partitions) still have unassigned weight. The algorithm should always have enough subtask capacity to place every key's weight; hitting this means the loop terminated early, which is treated as an internal invariant violation.

Source

Thrown at flink/v2.3/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapAssignment.java:175

        Maps.newHashMapWithExpectedSize(sortedStatistics.size());
    Iterator<SortKey> mapKeyIterator = sortedStatistics.keySet().iterator();
    int subtaskId = 0;
    SortKey currentKey = null;
    long keyRemainingWeight = 0L;
    long subtaskRemainingWeight = targetWeightPerSubtask;
    List<Integer> assignedSubtasks = Lists.newArrayList();
    List<Long> subtaskWeights = Lists.newArrayList();
    while (mapKeyIterator.hasNext() || currentKey != null) {
      // This should never happen because target weight is calculated using ceil function.
      if (subtaskId >= numPartitions) {
        LOG.error(
            "Internal algorithm error: exhausted subtasks with unassigned keys left. number of partitions: {}, "
                + "target weight per subtask: {}, close file cost in weight: {}, data statistics: {}",
            numPartitions,
            targetWeightPerSubtask,
            closeFileCostWeight,
            sortedStatistics);
        throw new IllegalStateException(
            "Internal algorithm error: exhausted subtasks with unassigned keys left");
      }

      if (currentKey == null) {
        currentKey = mapKeyIterator.next();
        keyRemainingWeight = sortedStatistics.get(currentKey);
      }

      assignedSubtasks.add(subtaskId);
      if (keyRemainingWeight < subtaskRemainingWeight) {
        // assign the remaining weight of the key to the current subtask
        subtaskWeights.add(keyRemainingWeight);
        subtaskRemainingWeight -= keyRemainingWeight;
        keyRemainingWeight = 0L;
      } else {
        // filled up the current subtask
        long assignedWeight = subtaskRemainingWeight;
        keyRemainingWeight -= subtaskRemainingWeight;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check the write parallelism (sink subtask count) — a pathologically small value relative to the number of partitions can starve the algorithm; increase it.
2. Inspect the logged diagnostics (numPartitions, targetWeightPerSubtask, closeFileCostWeight, sortedStatistics) for overflow or skew.
3. Increase write.task.max / sink parallelism so target weight per subtask is not degenerate.
4. If reproducible with normal inputs, report it as a bug with the logged statistics — it signals an algorithm defect, not user error.

Example fix

// before
env.from(...).sinkTo(icebergSink); // parallelism 1
// after
sinkBuilder = IcebergSink.builder()...;
env.from(...).sinkTo(icebergSink).setParallelism(numPartitions);

if (sinkParallelism < 1 || sinkParallelism > numPartitions * 4) {
  throw new IllegalArgumentException("Suspicious sink parallelism: " + sinkParallelism);
}

try {
  assignment = MapAssignment.buildAssignment(...);
} catch (IllegalStateException e) {
  LOG.error("Assignment algorithm failed; check logged diagnostics", e);
  throw e;
}

When it happens

Trigger: Invoked via assignment() during MapRangePartitioner planning when sorted key statistics are inconsistent with the computed targetWeightPerSubtask / closeFileCostWeight — e.g., extreme key-weight skew, integer overflow in weights, or numPartitions reduced to an unexpectedly small value.

Common situations: Very skewed data statistics where one or few partitions dominate weight; write parallelism drastically reduced between runs; a genuine bug in the weighting math surfaced by an unusual statistics distribution.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Internal algorithm error: exhausted subtasks with unassigned
• Invalid statistics type: %s. Should be Map or Sketch
• Internal algorithm error: exhausted subtasks with unassigned
• Unknown comparison result
• Unexpected entry status, not added or deleted: %s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0a5473d91db1372a. Report an issue: GitHub.