apache/druid · error · IllegalStateException

Stage does not shuffle

Error message

Stage does not shuffle

What it means

getShuffleSpec() returns the stage's shuffle spec, but stages that do not shuffle (e.g. leaf/ingestion stages) have a null shuffleSpec. Calling this accessor on such a stage throws IllegalStateException("Stage does not shuffle"), protecting callers from treating a non-shuffling stage as a shuffle participant.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/StageDefinition.java:296

  /**
   * Get a {@link SegmentPruner} from the {@link StageProcessor} for a given 'input number' from {@link #inputSpecs}.
   * This can be used to best-effort prune the set of {@link org.apache.druid.timeline.DataSegment} to process in order
   * to reduce the working set before processing begins
   */
  public SegmentPruner getSegmentPruner(int inputNumber)
  {
    return processor.getPruner(inputSpecs.get(inputNumber), inputNumber);
  }

  /**
   * Returns the {@link ShuffleSpec} for this stage, if {@link #doesShuffle()}.
   *
   * @throws IllegalStateException if this stage does not shuffle
   */
  public ShuffleSpec getShuffleSpec()
  {
    if (shuffleSpec == null) {
      throw new IllegalStateException("Stage does not shuffle");
    }

    return shuffleSpec;
  }

  /**
   * Returns the {@link ClusterBy} of the {@link ShuffleSpec} if set, otherwise {@link ClusterBy#none()}.
   */
  public ClusterBy getClusterBy()
  {
    if (shuffleSpec != null) {
      return shuffleSpec.clusterBy();
    } else {
      return ClusterBy.none();
    }
  }

  /**

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Guard with stageDefinition.getShuffleSpecForDiagnostics() or check mustGatherResultKeyStatistics()/shuffleSpec presence before calling
2. Only call getShuffleSpec() on stages where the builder set a shuffle spec
3. Restructure the traversal to skip non-shuffling stages (leaf and output stages)

Example fix

// before
ShuffleSpec spec = stage.getShuffleSpec();
// after
if (stage.getProcessor() instanceof etc) { /* leaf */ } else {
  ShuffleSpec spec = stage.getShuffleSpec();
}

boolean shuffles = stageDef.getProcessor() != null
    && stageDef.getShuffleSpecForDiagnostics() != null; // or check via builder inputs

// Narrow to stages that shuffle before calling the accessor
boolean hasShuffle(StageDefinition d) { return !d.mustGatherResultKeyStatistics() && d.getProcessor() != null; }

try { ShuffleSpec spec = d.getShuffleSpec(); } catch (IllegalStateException e) { /* stage is a leaf; skip */ }

When it happens

Trigger: Calling stageDefinition.getShuffleSpec() on a stage built without builder.shuffleSpec(...) — typically a source/leaf stage, or a final stage that writes output directly.

Common situations: Walking a QueryKit/stage graph generically and calling getShuffleSpec() on every stage without first checking shuffleSpec presence, or kernel/worker code assuming all stages shuffle.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Cannot shuffle with spec [%s] and nil clusterBy
• No statistics needed for stage[%d]
• unknown event type [%s]
• Unknown
• Unknown

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/39151f217a5c83a8. Report an issue: GitHub.