{"record":{"id":"2f7f9e29e71c02d7","repo":"apache/druid","slug":"statistics-gathered-but-not-required-for-stage-d","errorCode":null,"errorMessage":"Statistics gathered, but not required for stage[%d]","messagePattern":"Statistics gathered, but not required for stage\\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java","lineNumber":411,"sourceCode":"   *                      out of convenience.\n   */\n  public Either<Long, ClusterByPartitions> generatePartitionBoundariesForShuffle(\n      @Nullable ClusterByStatisticsCollector collector,\n      int maxPartitions\n  )\n  {\n    if (shuffleSpec == null) {\n      throw new ISE(\"No shuffle for stage[%d]\", getStageNumber());\n    } else if (shuffleSpec.kind() != ShuffleKind.GLOBAL_SORT) {\n      throw new ISE(\n          \"Shuffle of kind [%s] cannot generate partition boundaries for stage[%d]\",\n          shuffleSpec.kind(),\n          getStageNumber()\n      );\n    } else if (mustGatherResultKeyStatistics() && collector == null) {\n      throw new ISE(\"Statistics required, but not gathered for stage[%d]\", getStageNumber());\n    } else if (!mustGatherResultKeyStatistics() && collector != null) {\n      throw new ISE(\"Statistics gathered, but not required for stage[%d]\", getStageNumber());\n    } else {\n      return ((GlobalSortShuffleSpec) shuffleSpec).generatePartitionsForGlobalSort(collector, maxPartitions);\n    }\n  }\n\n  public ClusterByStatisticsCollector createResultKeyStatisticsCollector(\n      final FrameType frameType,\n      final int maxRetainedBytes\n  )\n  {\n    if (!mustGatherResultKeyStatistics()) {\n      throw new ISE(\"No statistics needed for stage[%d]\", getStageNumber());\n    }\n\n    return ClusterByStatisticsCollectorImpl.create(\n        shuffleSpec.clusterBy(),\n        signature,\n        frameType,","sourceCodeStart":393,"sourceCodeEnd":429,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java#L393-L429","documentation":"The mirror case of the previous check: if the stage does NOT require result key statistics but a non-null collector is passed to generatePartitionBoundariesForShuffle(), ISE(\"Statistics gathered, but not required for stage[%d]\") is thrown. The API enforces exact agreement between the stage's stats requirement and the caller's collected state.","triggerScenarios":"Calling generatePartitionBoundariesForShuffle(collector, maxPartitions) with a collector on a stage where mustGatherResultKeyStatistics() is false (no GLOBAL_SORT stats needed), e.g. reusing gathered statistics from a different stage.","commonSituations":"Kernel code carrying a collector from a previous stage into a stage that needs none, or tests copying the stats flow between stages incorrectly.","solutions":["Pass null as the collector for stages where mustGatherResultKeyStatistics() is false","Only pass the collector produced by this stage's createResultKeyStatisticsCollector","If statistics seem needed, fix the stage's shuffle spec so mustGatherResultKeyStatistics() returns true"],"exampleFix":"// before\nstageDef.generatePartitionBoundariesForShuffle(leftOverCollector, maxPartitions);\n// after\nstageDef.generatePartitionBoundariesForShuffle(\n    stageDef.mustGatherResultKeyStatistics() ? collector : null, maxPartitions);","handlingStrategy":"validation","validationCode":"ClusterByStatisticsCollector c = stageDef.mustGatherResultKeyStatistics() ? gathered : null;\nstageDef.generatePartitionBoundariesForShuffle(c, maxPartitions);","typeGuard":null,"tryCatchPattern":"try { d.generatePartitionBoundariesForShuffle(c, maxP); } catch (IllegalStateException e) { log.error(\"Unexpected stats state: {}\", e.getMessage()); }","preventionTips":["Pass null collector for stages that do not gather statistics","Never reuse collectors across stages","Keep stats flow aligned with mustGatherResultKeyStatistics()"],"tags":["msq","statistics","shuffle"],"backgroundTag":"invalid-argument-value","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}