apache/iceberg · error · IllegalStateException

Internal algorithm error: exhausted subtasks with unassigned

Error message

Internal algorithm error: exhausted subtasks with unassigned keys left

What it means

MapAssignment.buildAssignment distributes map-key weights across subtasks proportionally to data statistics. If the algorithm consumes all subtasks while keys with unassigned weight remain, it logs the algorithm parameters and throws IllegalStateException('Internal algorithm error: exhausted subtasks with unassigned keys left'). This is an invariant failure of the assignment algorithm, not a user configuration error per se.

Source

Thrown at flink/v1.20/flink/src/main/java/org/apache/iceberg/flink/sink/shuffle/MapAssignment.java:175

        Maps.newHashMapWithExpectedSize(sortedStatistics.size());
    Iterator<SortKey> mapKeyIterator = sortedStatistics.keySet().iterator();
    int subtaskId = 0;
    SortKey currentKey = null;
    long keyRemainingWeight = 0L;
    long subtaskRemainingWeight = targetWeightPerSubtask;
    List<Integer> assignedSubtasks = Lists.newArrayList();
    List<Long> subtaskWeights = Lists.newArrayList();
    while (mapKeyIterator.hasNext() || currentKey != null) {
      // This should never happen because target weight is calculated using ceil function.
      if (subtaskId >= numPartitions) {
        LOG.error(
            "Internal algorithm error: exhausted subtasks with unassigned keys left. number of partitions: {}, "
                + "target weight per subtask: {}, close file cost in weight: {}, data statistics: {}",
            numPartitions,
            targetWeightPerSubtask,
            closeFileCostWeight,
            sortedStatistics);
        throw new IllegalStateException(
            "Internal algorithm error: exhausted subtasks with unassigned keys left");
      }

      if (currentKey == null) {
        currentKey = mapKeyIterator.next();
        keyRemainingWeight = sortedStatistics.get(currentKey);
      }

      assignedSubtasks.add(subtaskId);
      if (keyRemainingWeight < subtaskRemainingWeight) {
        // assign the remaining weight of the key to the current subtask
        subtaskWeights.add(keyRemainingWeight);
        subtaskRemainingWeight -= keyRemainingWeight;
        keyRemainingWeight = 0L;
      } else {
        // filled up the current subtask
        long assignedWeight = subtaskRemainingWeight;
        keyRemainingWeight -= subtaskRemainingWeight;

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Increase the write (range-partition) parallelism so target weight per subtask is not degenerately small relative to total key weight
2. Inspect the logged values (numPartitions, targetWeightPerSubtask, closeFileCostInWeight) for zero/invalid values and fix the causing configuration
3. Refresh data statistics by running a few checkpoints so the assignment is recomputed from current distributions
4. Collect the full log context and file an Iceberg issue — this indicates an algorithm bug if inputs look sane

long totalWeight = sortedStatistics.values().stream().mapToLong(Long::longValue).sum();
Preconditions.checkState(numPartitions > 0 && totalWeight > 0, "Invalid inputs for map assignment: partitions=%s weight=%s", numPartitions, totalWeight);

try {
  MapAssignment a = MapAssignment.assignment(subtasks, closeFileCostWeight, stats);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("exhausted subtasks")) {
    // fall back to even distribution or recompute with fresh statistics
  }
  throw e;
}

When it happens

Trigger: During assignment computation, the loop runs out of subtask slots while sortedStatistics still holds keys with remaining weight — degenerate inputs such as zero/negative target weight per subtask, extreme weight distributions, or bad data statistics can drive the algorithm into this state.

Common situations: Very skewed key distributions combined with small write parallelism; closeFileCost weighting configurations producing a target weight per subtask that cannot cover remaining keys; write parallelism changed drastically (rescale) with stale statistics.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Internal algorithm error: exhausted subtasks with unassigned
• Invalid statistics type: %s. Should be Map or Sketch
• Internal algorithm error: exhausted subtasks with unassigned
• Illegal table name:
• Namespaces are not supported by catalog:

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/eda9aaf3c0495e58. Report an issue: GitHub.