apache/druid · error · IllegalArgumentException

Cannot shuffle with spec [%s] and nil clusterBy

Error message

Cannot shuffle with spec [%s] and nil clusterBy

What it means

The StageDefinition constructor validates that any stage whose shuffle spec requires result key statistics (mustGatherResultKeyStatistics()) has a non-empty clusterBy in its shuffle spec. A shuffle spec with an empty clusterBy cannot produce the partitioning keys the statistics collector needs, so IAE("Cannot shuffle with spec [%s] and nil clusterBy") is thrown while building the stage.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:138

    this.inputSpecs = Preconditions.checkNotNull(inputSpecs, "inputSpecs");

    if (broadcastInputNumbers == null) {
      this.broadcastInputNumbers = IntSets.emptySet();
    } else if (broadcastInputNumbers instanceof IntSet) {
      this.broadcastInputNumbers = (IntSet) broadcastInputNumbers;
    } else {
      this.broadcastInputNumbers = new IntAVLTreeSet(broadcastInputNumbers);
    }

    this.processor = Preconditions.checkNotNull(processor, "processor");
    this.signature = Preconditions.checkNotNull(signature, "signature");
    this.shuffleSpec = shuffleSpec;
    this.maxWorkerCount = maxWorkerCount;
    this.shuffleCheckHasMultipleValues = shuffleCheckHasMultipleValues;
    this.frameReader = Suppliers.memoize(() -> FrameReader.create(signature))::get;

    if (mustGatherResultKeyStatistics() && shuffleSpec.clusterBy().isEmpty()) {
      throw new IAE("Cannot shuffle with spec [%s] and nil clusterBy", shuffleSpec);
    }

    for (final String columnName : signature.getColumnNames()) {
      if (!signature.getColumnType(columnName).isPresent()) {
        throw new ISE("Missing type for column [%s]", columnName);
      }
    }

    for (final int broadcastInputNumber : this.broadcastInputNumbers) {
      if (broadcastInputNumber < 0 || broadcastInputNumber >= inputSpecs.size()) {
        throw new ISE("Broadcast input number out of range [%s]", broadcastInputNumber);
      }
    }
  }

  public static boolean mustGatherResultKeyStatistics(@Nullable final ShuffleSpec shuffleSpec)
  {
    return shuffleSpec != null

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Give the shuffle spec a non-empty ClusterBy column list (typically the sort/ordering columns)
2. Use a shuffle spec kind that does not require result key statistics if no clustering is intended
3. Inspect the spec printed in the message and compare with a working query's stage spec

Example fix

// before
builder.shuffleSpec(GlobalSortShuffleSpec.withEmptyClusterBy());
// after
builder.shuffleSpec(new GlobalSortShuffleSpec(ClusterBy.forColumns(List.of("__time"))));

if (mustGatherResultKeyStatistics(spec) && spec.clusterBy().isEmpty()) {
  throw new IllegalArgumentException("shuffle spec requires non-empty clusterBy");
}

try { StageDefinition def = builder.build(); } catch (IllegalArgumentException e) { log.error("Bad stage spec: {}", e.getMessage()); }

When it happens

Trigger: Constructing a StageDefinition via StageDefinition.builder()/process() with a shuffle spec whose clusterBy() returns empty, while the spec kind is one that requires gathering result key statistics (e.g. a global sort shuffle).

Common situations: Building MSQ stage graphs programmatically, custom query tooling emitting a global-sort shuffle with no clustering columns, or a query planner bug producing an inconsistent spec/clusterBy pair.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Cannot have empty worker set
• Can not supply empty segments as input, please use either nu
• Partition count must be at least 1
• Partition count must be 1 when adjustable is true, but was [
• ClusterBy key must be sortable

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ef9dccfd95c1b2d5. Report an issue: GitHub.