apache/druid · error · IllegalStateException

DimensionDistributionPhaseRunner has been stopped. %s

Error message

DimensionDistributionPhaseRunner has been stopped. %s

What it means

During the dimension-distribution phase of a range-partitioned parallel batch ingest, the runner checks whether a shutdown/stop has been requested before merging sub-task reports into partition boundaries. If stopGracefully() was called (due to an error elsewhere, supervisor cancellation, or task shutdown), it throws ISE reporting the stop reason.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialDimensionDistributionParallelIndexTaskRunner.java:122

    allReportsProcessedPhaser.register();
    executor.submit(() -> extractDistributionsFromReport(report));
  }

  /**
   * Map from an interval to PartitionBoundaries calculated by applying the target
   * row size on the final StringDistribution. The final distribution for an
   * interval is obtained by merging the distributions reported by all the
   * sub-tasks for that interval.
   */
  public Map<Interval, PartitionBoundaries> getIntervalToPartitionBoundaries(
      DimensionRangePartitionsSpec partitionsSpec
  )
  {
    waitToProcessPendingReports();

    // Do not proceed if a shutdown has been requested
    if (getStopReason() != null) {
      throw new ISE("DimensionDistributionPhaseRunner has been stopped. %s", getStopReason());
    }

    // Merge distributions only from succeeded sub-tasks
    final Set<String> succeededTaskIds = super.getReports().keySet();
    final Map<Interval, PartitionBoundaries> intervalToPartitions = new HashMap<>();
    intervalToTaskIds.forEach(
        (interval, subTaskIds) -> {
          final File intervalDir = getIntervalDistributionDir(interval);
          final StringDistributionMerger merger = new StringSketchMerger();
          subTaskIds
              .stream()
              .filter(succeededTaskIds::contains)
              .map(subTaskId -> readDistributionFromFile(intervalDir, subTaskId))
              .forEach(merger::merge);
          final StringDistribution mergedDistribution = merger.getResult();

          final PartitionBoundaries partitions;
          Integer targetRowsPerSegment = partitionsSpec.getTargetRowsPerSegment();

View on GitHub (pinned to 9b90983fd2)

Solutions

Read the stop reason embedded in the message to find the root failure that caused graceful shutdown, and fix that first.
Retry the batch ingestion task; this error usually follows an earlier failure, not an input-config problem.
Check Overlord task logs for the sub-task that failed or was killed during the distribution phase.

Defensive patterns

Strategy: try-catch

Try / catch

try { runner.getIntervalToPartitionBoundaries(...) } catch (IllegalStateException e) { if (e.getMessage().startsWith("DimensionDistributionPhaseRunner has been stopped")) { log.error("run aborted: {}", e.getMessage()); /* retry whole supervisor task */ } else { throw e; } }

Prevention

Treat this as a downstream symptom: always investigate the earlier stop reason/root failure.
Avoid killing supervisor tasks mid-distribution-phase; cancel cleanly between phases.
Ensure worker disk space is sufficient so earlier phases don't fail and trigger graceful stop.

When it happens

Trigger: Calling getIntervalToPartitionBoundaries() after stopGracefully() was invoked on the runner - typically because a sibling sub-task failed and wrote a distribution file error, or the supervisor task was asked to stop.

Common situations: A multi-phase range partitioned batch job aborting mid-flight; one sub-task's distribution write failed causing graceful stop; operator killing the supervisor task while distribution collection was pending.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/b435299968886c64. Report an issue: GitHub.