{"record":{"id":"b95d976fd4b944ba","repo":"apache/druid","slug":"statistics-required-but-not-gathered-for-stage-d","errorCode":null,"errorMessage":"Statistics required, but not gathered for stage[%d]","messagePattern":"Statistics required, but not gathered for stage\\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java","lineNumber":409,"sourceCode":"   *                      is only used when the number of partitions is determined ahead of time by the\n   *                      {@link ShuffleSpec}, so {@link Integer#MAX_VALUE} is typically provided for this parameter\n   *                      out of convenience.\n   */\n  public Either<Long, ClusterByPartitions> generatePartitionBoundariesForShuffle(\n      @Nullable ClusterByStatisticsCollector collector,\n      int maxPartitions\n  )\n  {\n    if (shuffleSpec == null) {\n      throw new ISE(\"No shuffle for stage[%d]\", getStageNumber());\n    } else if (shuffleSpec.kind() != ShuffleKind.GLOBAL_SORT) {\n      throw new ISE(\n          \"Shuffle of kind [%s] cannot generate partition boundaries for stage[%d]\",\n          shuffleSpec.kind(),\n          getStageNumber()\n      );\n    } else if (mustGatherResultKeyStatistics() && collector == null) {\n      throw new ISE(\"Statistics required, but not gathered for stage[%d]\", getStageNumber());\n    } else if (!mustGatherResultKeyStatistics() && collector != null) {\n      throw new ISE(\"Statistics gathered, but not required for stage[%d]\", getStageNumber());\n    } else {\n      return ((GlobalSortShuffleSpec) shuffleSpec).generatePartitionsForGlobalSort(collector, maxPartitions);\n    }\n  }\n\n  public ClusterByStatisticsCollector createResultKeyStatisticsCollector(\n      final FrameType frameType,\n      final int maxRetainedBytes\n  )\n  {\n    if (!mustGatherResultKeyStatistics()) {\n      throw new ISE(\"No statistics needed for stage[%d]\", getStageNumber());\n    }\n\n    return ClusterByStatisticsCollectorImpl.create(\n        shuffleSpec.clusterBy(),","sourceCodeStart":391,"sourceCodeEnd":427,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java#L391-L427","documentation":"For GLOBAL_SORT stages that require result key statistics (mustGatherResultKeyStatistics() is true), generatePartitionBoundariesForShuffle() needs the ClusterByStatisticsCollector gathered by workers. Passing null collector for such a stage throws ISE(\"Statistics required, but not gathered for stage[%d]\").","triggerScenarios":"Calling generatePartitionBoundariesForShuffle(null, maxPartitions) on a global-sort stage whose mustGatherResultKeyStatistics() returns true — i.e. the kernel/controller never gathered statistics (createResultKeyStatisticsCollector was not run or results were lost).","commonSituations":"Controller logic that skips the statistics-gathering phase (e.g. when worker count is 1 and statistics are assumed unnecessary but the spec still requires them), or tests passing null directly.","solutions":["Gather statistics first: call createResultKeyStatisticsCollector on workers and pass the resulting collector","Ensure the controller reads the statistics snapshot and passes a non-null collector for stats-required stages","Verify the stage genuinely needs statistics; if not, fix mustGatherResultKeyStatistics/shuffleSpec so the pair is consistent"],"exampleFix":"// before\nstageDef.generatePartitionBoundariesForShuffle(null, maxPartitions);\n// after\nstageDef.generatePartitionBoundariesForShuffle(gatheredCollector, maxPartitions);","handlingStrategy":"validation","validationCode":"if (stageDef.mustGatherResultKeyStatistics() && collector == null) {\n  throw new IllegalStateException(\"statistics must be gathered before boundary generation\");\n}","typeGuard":null,"tryCatchPattern":"try { d.generatePartitionBoundariesForShuffle(collector, maxP); } catch (IllegalStateException e) { log.error(\"Missing stats: {}\", e.getMessage()); }","preventionTips":["Always run the statistics-gathering phase before generating boundaries","Pass the collector produced by this stage's createResultKeyStatisticsCollector","Verify controller reads statistics snapshots for global-sort stages"],"tags":["msq","statistics","shuffle"],"backgroundTag":"missing-required-argument","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}