{"record":{"id":"ef9dccfd95c1b2d5","repo":"apache/druid","slug":"cannot-shuffle-with-spec-s-and-nil-clusterby","errorCode":null,"errorMessage":"Cannot shuffle with spec [%s] and nil clusterBy","messagePattern":"Cannot shuffle with spec \\[(.+?)\\] and nil clusterBy","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java","lineNumber":138,"sourceCode":"    this.inputSpecs = Preconditions.checkNotNull(inputSpecs, \"inputSpecs\");\n\n    if (broadcastInputNumbers == null) {\n      this.broadcastInputNumbers = IntSets.emptySet();\n    } else if (broadcastInputNumbers instanceof IntSet) {\n      this.broadcastInputNumbers = (IntSet) broadcastInputNumbers;\n    } else {\n      this.broadcastInputNumbers = new IntAVLTreeSet(broadcastInputNumbers);\n    }\n\n    this.processor = Preconditions.checkNotNull(processor, \"processor\");\n    this.signature = Preconditions.checkNotNull(signature, \"signature\");\n    this.shuffleSpec = shuffleSpec;\n    this.maxWorkerCount = maxWorkerCount;\n    this.shuffleCheckHasMultipleValues = shuffleCheckHasMultipleValues;\n    this.frameReader = Suppliers.memoize(() -> FrameReader.create(signature))::get;\n\n    if (mustGatherResultKeyStatistics() && shuffleSpec.clusterBy().isEmpty()) {\n      throw new IAE(\"Cannot shuffle with spec [%s] and nil clusterBy\", shuffleSpec);\n    }\n\n    for (final String columnName : signature.getColumnNames()) {\n      if (!signature.getColumnType(columnName).isPresent()) {\n        throw new ISE(\"Missing type for column [%s]\", columnName);\n      }\n    }\n\n    for (final int broadcastInputNumber : this.broadcastInputNumbers) {\n      if (broadcastInputNumber < 0 || broadcastInputNumber >= inputSpecs.size()) {\n        throw new ISE(\"Broadcast input number out of range [%s]\", broadcastInputNumber);\n      }\n    }\n  }\n\n  public static boolean mustGatherResultKeyStatistics(@Nullable final ShuffleSpec shuffleSpec)\n  {\n    return shuffleSpec != null","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java#L120-L156","documentation":"The StageDefinition constructor validates that any stage whose shuffle spec requires result key statistics (mustGatherResultKeyStatistics()) has a non-empty clusterBy in its shuffle spec. A shuffle spec with an empty clusterBy cannot produce the partitioning keys the statistics collector needs, so IAE(\"Cannot shuffle with spec [%s] and nil clusterBy\") is thrown while building the stage.","triggerScenarios":"Constructing a StageDefinition via StageDefinition.builder()/process() with a shuffle spec whose clusterBy() returns empty, while the spec kind is one that requires gathering result key statistics (e.g. a global sort shuffle).","commonSituations":"Building MSQ stage graphs programmatically, custom query tooling emitting a global-sort shuffle with no clustering columns, or a query planner bug producing an inconsistent spec/clusterBy pair.","solutions":["Give the shuffle spec a non-empty ClusterBy column list (typically the sort/ordering columns)","Use a shuffle spec kind that does not require result key statistics if no clustering is intended","Inspect the spec printed in the message and compare with a working query's stage spec"],"exampleFix":"// before\nbuilder.shuffleSpec(GlobalSortShuffleSpec.withEmptyClusterBy());\n// after\nbuilder.shuffleSpec(new GlobalSortShuffleSpec(ClusterBy.forColumns(List.of(\"__time\"))));","handlingStrategy":"validation","validationCode":"if (mustGatherResultKeyStatistics(spec) && spec.clusterBy().isEmpty()) {\n  throw new IllegalArgumentException(\"shuffle spec requires non-empty clusterBy\");\n}","typeGuard":null,"tryCatchPattern":"try { StageDefinition def = builder.build(); } catch (IllegalArgumentException e) { log.error(\"Bad stage spec: {}\", e.getMessage()); }","preventionTips":["Always attach clustering columns to global-sort shuffle specs","Validate shuffleSpec/clusterBy consistency in stage-graph builders","Compare with stage specs generated by a working query"],"tags":["msq","stage-definition","validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}