apache/druid · error · IllegalStateException

Output partitions are not ready yet

Error message

Output partitions are not ready yet

What it means

getOutputPartitions() reads the partitioning scheme, but only after it has been computed by the partitioning phase. If outputPartitionsFuture is not yet done, it throws IllegalStateException — callers must not read output partitions before the partition-discovery step of the sort completes. It is an internal timing/contract guard: waiting should be done via the future, not by polling this method.

Solutions

1. Await outputPartitionsFuture (or the future returned by run()) before reading output partitions; do not poll getOutputPartitions().
2. Hook the completion callback: read partitions only after the partition-determination phase future completes (as run() does via its listener).
3. If this occurs inside stock Druid code paths, verify no custom extension altered the ordering of runMerger/setTotalMergingLevelsIfPossible calls.
4. Upgrade Druid if hit spontaneously — this can indicate an internal scheduling regression fixed in later releases.

Example fix

// before
int partitions = superSorter.outputPartitionCount(); // may throw if partitions unknown yet
// after
ListenableFuture<ClusterByPartitions> partitionsFuture = superSorter.outputPartitionsFuture();
ClusterByPartitions partitions = FutureUtils.getUnchecked(partitionsFuture, true);

if (!outputPartitionsFuture.isDone()) {
  // wait instead of reading
  ClusterByPartitions partitions = FutureUtils.getUnchecked(outputPartitionsFuture, true);
}

static boolean outputPartitionsReady(ListenableFuture<ClusterByPartitions> f) {
  return f.isDone();
}

try {
  partitions = superSorter.outputPartitionCount();
} catch (IllegalStateException e) {
  partitions = FutureUtils.getUnchecked(partitionsFuture, true).size();
}

When it happens

Trigger: Calling getOutputPartitions() (directly or indirectly via partitions(), outputPartitionCount(), or runNextUltimateMerger()) while outputPartitionsFuture is still pending — e.g. invoking outputPartitionCount() right after run() before the partition-determining processors finish, or wiring an ultimate merger before partition info resolves.

Common situations: Custom code extending or instrumenting SuperSorter that reads partition info too early; debugging/instrumentation logging partition counts immediately after starting the sorter; race conditions in modified scheduling code that skips the future-completion callback that normally triggers these reads.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Attempt to add row to swapped-out sink for segment
• Background lookup manager thread could not be cancelled
• Cache reference is null
• Can't find surrogate task
• can't start.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/4a2a55dcdaec40e6. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/frame/processor/SuperSorter.java:908

      } else {
        // Use two levels: no need to have a partitioned penultimate layer.
        totalMergingLevels = 2;
      }
    } else {
      totalMergingLevels = level + 1;
    }

    for (int i = level; i < totalMergingLevels; i++) {
      superSorterProgressTracker.setTotalMergersForLevel(i, 1);
    }

    superSorterProgressTracker.setTotalMergingLevels(totalMergingLevels);
  }

  private ClusterByPartitions getOutputPartitions()
  {
    if (!outputPartitionsFuture.isDone()) {
      throw new ISE("Output partitions are not ready yet");
    }

    return FutureUtils.getUnchecked(outputPartitionsFuture, true);
  }

  @GuardedBy("runWorkersLock")
  private long getTotalMergersInLevel(final int level)
  {
    if (totalInputFrames == UNKNOWN_TOTAL || totalMergingLevels == UNKNOWN_LEVEL) {
      return UNKNOWN_TOTAL;
    } else if (level >= totalMergingLevels) {
      throw new ISE("Invalid level %d", level);
    } else if (level == totalMergingLevels - 1) {
      if (outputPartitionsFuture.isDone()) {
        return totalInputFrames == 0 ? 0 : getOutputPartitions().size();
      } else {
        return UNKNOWN_TOTAL;
      }

View on GitHub (pinned to 9b90983fd2)