apache/druid · error · IllegalStateException

Total merging levels already defined for the merge sort.

Error message

Total merging levels already defined for the merge sort.

What it means

SuperSorterProgressTracker.setTotalMergingLevels throws this when totalMergingLevels is already set and someone tries to set it again. The tracker treats the total merging level count as a write-once property of the sort, so a second call indicates the sorter is being driven twice or by two different components. It is a synchronization/initialization-order bug indicator.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/processor/SuperSorterProgressTracker.java:91

      Collections.emptyMap(),
      Collections.emptyMap(),
      SuperSorter.UNKNOWN_TOTAL,
      true
  );

  public SuperSorterProgressTracker()
  {
    this.levelToMergedBatches = new HashMap<>();
    this.levelToTotalBatches = new HashMap<>();
  }

  /**
   * Set total merging levels for the SuperSorter it is tracking. Can be set only once
   */
  public synchronized void setTotalMergingLevels(final int totalMergingLevels)
  {
    if (this.totalMergingLevels != SuperSorter.UNKNOWN_LEVEL) {
      throw new ISE("Total merging levels already defined for the merge sort.");
    }
    levelToMergedBatches.keySet().stream().max(Ordering.natural()).ifPresent(max -> {
      if (max >= totalMergingLevels) {
        throw new ISE(
            "Max level found in levelToMergedBatches is %d (0-indexed). Cannot set totalMergingLevels to %d",
            max,
            totalMergingLevels
        );
      }
    });
    levelToTotalBatches.keySet().stream().max(Ordering.natural()).ifPresent(max -> {
      if (max >= totalMergingLevels) {
        throw new ISE(
            "Max level found in levelToTotalBatches is %d (0-indexed). Cannot set totalMergingLevels to %d",
            max,
            totalMergingLevels
        );
      }

View on GitHub (pinned to 9b90983fd2)

Solutions

Set totalMergingLevels exactly once, before any merge activity; make the setting call idempotent on the caller side (check SuperSorter.UNKNOWN_LEVEL first).
Ensure each SuperSorter instance has its own dedicated SuperSorterProgressTracker; don't share trackers across sorters or retries.
If re-running a sort, construct a fresh tracker instead of reusing the previous instance.
Verify no code path (e.g., a partition-count update) recomputes and re-sets levels mid-sort.

Example fix

// before
tracker.setTotalMergingLevels(levels); // may run twice
// after
if (tracker.getTotalMergingLevels() == SuperSorter.UNKNOWN_LEVEL) {
  tracker.setTotalMergingLevels(levels);
}

Defensive patterns

Strategy: validation

Validate before calling

if (tracker.getTotalMergingLevels() == SuperSorter.UNKNOWN_LEVEL) {
  tracker.setTotalMergingLevels(levels);
}

Try / catch

try {
  tracker.setTotalMergingLevels(levels);
} catch (IllegalStateException e) {
  if (!e.getMessage().contains("already defined")) {
    throw e;
  } // already set — acceptable if the value matches
}

Prevention

Give every SuperSorter its own tracker instance.
Create a fresh tracker for each retry or re-run.
Set the level count exactly once, at sort initialization, before any merge activity.

When it happens

Trigger: Calling setTotalMergingLevels twice on the same tracker instance — e.g., both a direct-merger setup and a middle-merger setup path compute and set the level count, or a sorter is reinitialized reusing the old tracker.

Common situations: Developers wiring SuperSorter with a shared tracker across two sorting phases; retry logic that re-runs the initialization block; unit tests reusing a tracker fixture without resetting it.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/65a38bab060db4ad. Report an issue: GitHub.