apache/druid · error · IllegalStateException

Max level found in levelToMergedBatches is %d (0-indexed). C

Error message

Max level found in levelToMergedBatches is %d (0-indexed). Cannot set totalMergingLevels to %d

What it means

In setTotalMergingLevels, after confirming the level count is unset, the tracker validates existing batch bookkeeping: if levelToMergedBatches already contains a level index >= the proposed totalMergingLevels, setting the count would invalidate those records, so it throws. The recorded merged-batch levels must fit inside the new total level range (0-indexed).

Source

Thrown at processing/src/main/java/org/apache/druid/frame/processor/SuperSorterProgressTracker.java:95

  );

  public SuperSorterProgressTracker()
  {
    this.levelToMergedBatches = new HashMap<>();
    this.levelToTotalBatches = new HashMap<>();
  }

  /**
   * Set total merging levels for the SuperSorter it is tracking. Can be set only once
   */
  public synchronized void setTotalMergingLevels(final int totalMergingLevels)
  {
    if (this.totalMergingLevels != SuperSorter.UNKNOWN_LEVEL) {
      throw new ISE("Total merging levels already defined for the merge sort.");
    }
    levelToMergedBatches.keySet().stream().max(Ordering.natural()).ifPresent(max -> {
      if (max >= totalMergingLevels) {
        throw new ISE(
            "Max level found in levelToMergedBatches is %d (0-indexed). Cannot set totalMergingLevels to %d",
            max,
            totalMergingLevels
        );
      }
    });
    levelToTotalBatches.keySet().stream().max(Ordering.natural()).ifPresent(max -> {
      if (max >= totalMergingLevels) {
        throw new ISE(
            "Max level found in levelToTotalBatches is %d (0-indexed). Cannot set totalMergingLevels to %d",
            max,
            totalMergingLevels
        );
      }
    });

    this.totalMergingLevels = totalMergingLevels;
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Always call setTotalMergingLevels before recording any merged batches so levelToMergedBatches is empty at set time.
2. Ensure the computed totalMergingLevels accounts for the maximum level actually in use by the merge scheduler.
3. If partition/level counts changed, create a fresh tracker rather than mutating the existing one.
4. Audit callers that update progress with level indices and confirm they derive the index from the same totalMergingLevels value.

Example fix

// before
tracker.updateMergedBatches(level, batches); // level from old plan
tracker.setTotalMergingLevels(newSmallerLevels); // throws
// after
tracker.setTotalMergingLevels(levels); // set first
tracker.updateMergedBatches(level, batches);

// Ensure no merged-batch records exist before setting levels
assert tracker.getLevelToMergedBatches().isEmpty() : "set totalMergingLevels before tracking batches";

When it happens

Trigger: Progress was already tracked (updateMergedBatches) with level indices derived from an assumed or previous level count, then setTotalMergingLevels is called with a smaller value; typically occurs when merge work started before the level count was finalized.

Common situations: A sorter whose partition count shrinks between planning and execution (dynamic partition changes), causing previously recorded levels to exceed the new total; or unit tests that pre-populate batch progress before setting levels.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Max level found in levelToTotalBatches is %d (0-indexed). Ca
• Invalid level %d
• Unable to set %d total mergers for level %d. Level must be n
• Cannot set total mergers for level %d. Valid levels range fr
• Total mergers are already present for the level %d

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/6bad13229d0a35c4. Report an issue: GitHub.