apache/iceberg · error · IllegalStateException

Internal algorithm error: exhausted subtasks with unassigned

Error message

Internal algorithm error: exhausted subtasks with unassigned keys left

What it means

MapAssignment.buildAssignment distributes map keys across subtasks by weight; the algorithm should always be able to place remaining key weight on unassigned subtasks. Reaching this IllegalStateException means the iteration exhausted subtasks while keys with unassigned weight remained — an internal invariant violation in the assignment algorithm.

Source

Thrown at flink/v2.1/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapAssignment.java:175

        Maps.newHashMapWithExpectedSize(sortedStatistics.size());
    Iterator<SortKey> mapKeyIterator = sortedStatistics.keySet().iterator();
    int subtaskId = 0;
    SortKey currentKey = null;
    long keyRemainingWeight = 0L;
    long subtaskRemainingWeight = targetWeightPerSubtask;
    List<Integer> assignedSubtasks = Lists.newArrayList();
    List<Long> subtaskWeights = Lists.newArrayList();
    while (mapKeyIterator.hasNext() || currentKey != null) {
      // This should never happen because target weight is calculated using ceil function.
      if (subtaskId >= numPartitions) {
        LOG.error(
            "Internal algorithm error: exhausted subtasks with unassigned keys left. number of partitions: {}, "
                + "target weight per subtask: {}, close file cost in weight: {}, data statistics: {}",
            numPartitions,
            targetWeightPerSubtask,
            closeFileCostWeight,
            sortedStatistics);
        throw new IllegalStateException(
            "Internal algorithm error: exhausted subtasks with unassigned keys left");
      }

      if (currentKey == null) {
        currentKey = mapKeyIterator.next();
        keyRemainingWeight = sortedStatistics.get(currentKey);
      }

      assignedSubtasks.add(subtaskId);
      if (keyRemainingWeight < subtaskRemainingWeight) {
        // assign the remaining weight of the key to the current subtask
        subtaskWeights.add(keyRemainingWeight);
        subtaskRemainingWeight -= keyRemainingWeight;
        keyRemainingWeight = 0L;
      } else {
        // filled up the current subtask
        long assignedWeight = subtaskRemainingWeight;
        keyRemainingWeight -= subtaskRemainingWeight;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. File a bug with the logged diagnostics (numPartitions, targetWeightPerSubtask, closeFileCostWeight, statistics).
2. Disable the adaptive/shuffle-based write distribution (use write.distribution-mode=none or hash) as a workaround.
3. Upgrade the connector to a version with assignment algorithm fixes.

Example fix

// workaround: avoid the adaptive range shuffle
ALTER TABLE t UNSET TBLPROPERTIES ('write.distribution-mode');

try {
  MapAssignment a = MapAssignment.assignment(numPartitions, closeFileCostWeight, statistics);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("exhausted subtasks")) {
    LOG.error("Assignment algorithm bug; falling back to round-robin partitioning", e);
  } else throw e;
}

When it happens

Trigger: Hitting the weight-distribution loop with degenerate statistics inputs (e.g. extreme weight ratios, zero/negative target weight per subtask derived from closeFileCostWeight or huge key counts).

Common situations: Pathological collected data statistics during adaptive range shuffle (extremely skewed keys or tiny target weights); normally indicates a connector algorithm bug.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Internal algorithm error: exhausted subtasks with unassigned
• Invalid operator event type: ${eventType}
• Invalid statistics type: ${type}. Should be Map or Sketch
• Internal algorithm error: exhausted subtasks with unassigned
• Internal algorithm error: exhausted subtasks with unassigned

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/25580c961cace0c1. Report an issue: GitHub.