apache/druid · error · IllegalStateException

Statistics gathered, but not required for stage[%d]

Error message

Statistics gathered, but not required for stage[%d]

What it means

The mirror case of the previous check: if the stage does NOT require result key statistics but a non-null collector is passed to generatePartitionBoundariesForShuffle(), ISE("Statistics gathered, but not required for stage[%d]") is thrown. The API enforces exact agreement between the stage's stats requirement and the caller's collected state.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:411

   *                      out of convenience.
   */
  public Either<Long, ClusterByPartitions> generatePartitionBoundariesForShuffle(
      @Nullable ClusterByStatisticsCollector collector,
      int maxPartitions
  )
  {
    if (shuffleSpec == null) {
      throw new ISE("No shuffle for stage[%d]", getStageNumber());
    } else if (shuffleSpec.kind() != ShuffleKind.GLOBAL_SORT) {
      throw new ISE(
          "Shuffle of kind [%s] cannot generate partition boundaries for stage[%d]",
          shuffleSpec.kind(),
          getStageNumber()
      );
    } else if (mustGatherResultKeyStatistics() && collector == null) {
      throw new ISE("Statistics required, but not gathered for stage[%d]", getStageNumber());
    } else if (!mustGatherResultKeyStatistics() && collector != null) {
      throw new ISE("Statistics gathered, but not required for stage[%d]", getStageNumber());
    } else {
      return ((GlobalSortShuffleSpec) shuffleSpec).generatePartitionsForGlobalSort(collector, maxPartitions);
    }
  }

  public ClusterByStatisticsCollector createResultKeyStatisticsCollector(
      final FrameType frameType,
      final int maxRetainedBytes
  )
  {
    if (!mustGatherResultKeyStatistics()) {
      throw new ISE("No statistics needed for stage[%d]", getStageNumber());
    }

    return ClusterByStatisticsCollectorImpl.create(
        shuffleSpec.clusterBy(),
        signature,
        frameType,

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Pass null as the collector for stages where mustGatherResultKeyStatistics() is false
2. Only pass the collector produced by this stage's createResultKeyStatisticsCollector
3. If statistics seem needed, fix the stage's shuffle spec so mustGatherResultKeyStatistics() returns true

Example fix

// before
stageDef.generatePartitionBoundariesForShuffle(leftOverCollector, maxPartitions);
// after
stageDef.generatePartitionBoundariesForShuffle(
    stageDef.mustGatherResultKeyStatistics() ? collector : null, maxPartitions);

ClusterByStatisticsCollector c = stageDef.mustGatherResultKeyStatistics() ? gathered : null;
stageDef.generatePartitionBoundariesForShuffle(c, maxPartitions);

try { d.generatePartitionBoundariesForShuffle(c, maxP); } catch (IllegalStateException e) { log.error("Unexpected stats state: {}", e.getMessage()); }

When it happens

Trigger: Calling generatePartitionBoundariesForShuffle(collector, maxPartitions) with a collector on a stage where mustGatherResultKeyStatistics() is false (no GLOBAL_SORT stats needed), e.g. reusing gathered statistics from a different stage.

Common situations: Kernel code carrying a collector from a previous stage into a stage that needs none, or tests copying the stats flow between stages incorrectly.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Statistics required, but not gathered for stage[%d]
• No shuffle for stage[%d]
• Shuffle of kind [%s] cannot generate partition boundaries fo
• No statistics needed for stage[%d]
• Result partition information is not relevant to this stage b

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/2f7f9e29e71c02d7. Report an issue: GitHub.