apache/druid · error · IllegalStateException

Improper cleanup

Error message

Improper cleanup

What it means

SuperSorter.cleanUp() throws 'Improper cleanup' when cleanup is invoked while the sorter is not fully done (isAllDone() false) or while frame processors are still active (activeProcessors != 0). These two conditions together indicate a lifecycle-management bug: cleanup must only run after all merge work has completed and all processors have been released. It is a defensive assertion protecting against resource-management mistakes in the sorter's state machine.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/processor/SuperSorter.java:991

   *
   * Note: it is possible for this method to return true even when {@link #activeProcessors} is nonzero. Processors
   * take some time to exit after the instance becomes "done".
   */
  @GuardedBy("runWorkersLock")
  private boolean isAllDone()
  {
    return allDone.isDone() || allDone.isCancelled();
  }

  /**
   * Cleanup that must happen regardless of success or failure.
   */
  @GuardedBy("runWorkersLock")
  private void cleanUp()
  {
    if (!isAllDone() || activeProcessors != 0) {
      // This condition indicates a logic bug.
      throw new ISE("Improper cleanup");
    }

    if (log.isDebugEnabled()) {
      log.debug(stateString());
    }

    outputsReadyByLevel.clear();
    inputBuffer.clear();
    for (Map.Entry<String, PartitionedOutputChannel> cleanupEntry :
        levelAndRankToReadableChannelMap.entrySet()) {
      try {
        cleanupEntry.getValue().getReadableChannelSupplier().get().close();
      }
      catch (IOException e) {
        throw new UncheckedIOException("Unable to close channel for name : " + cleanupEntry.getKey(), e);
      }
    }
    levelAndRankToReadableChannelMap.clear();

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Ensure the sorter is fully drained (all processors done) before calling cleanUp(); call close()/cancel paths that stop processors and await their completion first.
2. Verify no double-cleanup: track whether cleanUp() already ran and guard against a second invocation.
3. Inspect processor exception handling — a processor that threw may never have decremented activeProcessors; make sure failure paths also release processor slots.
4. Capture stateString() (logged at debug) to see the sorter's level/channel state and identify which processor or channel never finished.

Example fix

// before
sorter.cleanUp(); // called immediately on error, processors still active
// after
if (sorter.isAllDone() && sorter.getActiveProcessors() == 0) {
  sorter.cleanUp();
} else {
  processorManager.cancelAndAwait();
  sorter.cleanUp();
}

if (sorter.isAllDone()) {
  sorter.cleanUp();
}

try {
  sorter.cleanUp();
} catch (IllegalStateException e) {
  if ("Improper cleanup".equals(e.getMessage())) {
    log.warn(e, "Sorter cleanup raced with active processors; state=%s", sorter.stateString());
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling cleanUp() before every merger processor has finished, or while some input/output processors still hold a reference (activeProcessors > 0); e.g., an error path calls cleanUp() while a worker thread is still running, or close() is invoked twice with interleaved processor completion.

Common situations: Hit during query cancellation or error teardown in the MSQ/frame-processor stack when the cancellation path races with ongoing merges; also when developing custom processors that fail to signal completion before triggering sorter cleanup.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• can't stop.
• can't start
• can't stop
• Emit called unexpectedly before service start
• Can't start.

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/793e963078969ebc. Report an issue: GitHub.