apache/druid · error · IllegalStateException

Statistics required, but not gathered for stage[%d]

Error message

Statistics required, but not gathered for stage[%d]

What it means

For GLOBAL_SORT stages that require result key statistics (mustGatherResultKeyStatistics() is true), generatePartitionBoundariesForShuffle() needs the ClusterByStatisticsCollector gathered by workers. Passing null collector for such a stage throws ISE("Statistics required, but not gathered for stage[%d]").

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:409

   *                      is only used when the number of partitions is determined ahead of time by the
   *                      {@link ShuffleSpec}, so {@link Integer#MAX_VALUE} is typically provided for this parameter
   *                      out of convenience.
   */
  public Either<Long, ClusterByPartitions> generatePartitionBoundariesForShuffle(
      @Nullable ClusterByStatisticsCollector collector,
      int maxPartitions
  )
  {
    if (shuffleSpec == null) {
      throw new ISE("No shuffle for stage[%d]", getStageNumber());
    } else if (shuffleSpec.kind() != ShuffleKind.GLOBAL_SORT) {
      throw new ISE(
          "Shuffle of kind [%s] cannot generate partition boundaries for stage[%d]",
          shuffleSpec.kind(),
          getStageNumber()
      );
    } else if (mustGatherResultKeyStatistics() && collector == null) {
      throw new ISE("Statistics required, but not gathered for stage[%d]", getStageNumber());
    } else if (!mustGatherResultKeyStatistics() && collector != null) {
      throw new ISE("Statistics gathered, but not required for stage[%d]", getStageNumber());
    } else {
      return ((GlobalSortShuffleSpec) shuffleSpec).generatePartitionsForGlobalSort(collector, maxPartitions);
    }
  }

  public ClusterByStatisticsCollector createResultKeyStatisticsCollector(
      final FrameType frameType,
      final int maxRetainedBytes
  )
  {
    if (!mustGatherResultKeyStatistics()) {
      throw new ISE("No statistics needed for stage[%d]", getStageNumber());
    }

    return ClusterByStatisticsCollectorImpl.create(
        shuffleSpec.clusterBy(),

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Gather statistics first: call createResultKeyStatisticsCollector on workers and pass the resulting collector
2. Ensure the controller reads the statistics snapshot and passes a non-null collector for stats-required stages
3. Verify the stage genuinely needs statistics; if not, fix mustGatherResultKeyStatistics/shuffleSpec so the pair is consistent

Example fix

// before
stageDef.generatePartitionBoundariesForShuffle(null, maxPartitions);
// after
stageDef.generatePartitionBoundariesForShuffle(gatheredCollector, maxPartitions);

if (stageDef.mustGatherResultKeyStatistics() && collector == null) {
  throw new IllegalStateException("statistics must be gathered before boundary generation");
}

try { d.generatePartitionBoundariesForShuffle(collector, maxP); } catch (IllegalStateException e) { log.error("Missing stats: {}", e.getMessage()); }

When it happens

Trigger: Calling generatePartitionBoundariesForShuffle(null, maxPartitions) on a global-sort stage whose mustGatherResultKeyStatistics() returns true — i.e. the kernel/controller never gathered statistics (createResultKeyStatisticsCollector was not run or results were lost).

Common situations: Controller logic that skips the statistics-gathering phase (e.g. when worker count is 1 and statistics are assumed unnecessary but the spec still requires them), or tests passing null directly.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• Statistics gathered, but not required for stage[%d]
• No shuffle for stage[%d]
• Shuffle of kind [%s] cannot generate partition boundaries fo
• No statistics needed for stage[%d]
• Result partition information is not relevant to this stage b

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/b95d976fd4b944ba. Report an issue: GitHub.