apache/druid · error · IllegalArgumentException

maxConcurrentStagesPerWorker must be >= 2 when pipelining

Error message

maxConcurrentStagesPerWorker must be >= 2 when pipelining

What it means

When pipelining is enabled, the kernel must run at least two stages concurrently per worker, so the constructor rejects maxConcurrentStages < 2 with IAE. Pipelining with only one concurrent stage slot is contradictory and cannot be scheduled.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java:66

  ControllerQueryKernelConfig(
      int maxRetainedPartitionSketchBytes,
      int maxConcurrentStages,
      boolean pipeline,
      boolean durableStorage,
      boolean faultTolerance,
      MSQDestination destination,
      @Nullable String controllerHost,
      @Nullable List<String> workerIds,
      Map<String, Object> workerContextMap
  )
  {
    if (maxRetainedPartitionSketchBytes <= 0) {
      throw new IAE("maxRetainedPartitionSketchBytes must be positive");
    }

    if (pipeline && maxConcurrentStages < 2) {
      throw new IAE("maxConcurrentStagesPerWorker must be >= 2 when pipelining");
    }

    if (maxConcurrentStages <= 0) {
      throw new IAE("maxConcurrentStagesPerWorker must be positive");
    }

    if (pipeline && faultTolerance) {
      throw new IAE("Cannot pipeline with fault tolerance");
    }

    if (pipeline && durableStorage) {
      throw new IAE("Cannot pipeline with durable storage");
    }

    if (faultTolerance && !durableStorage) {
      throw new IAE("Cannot have fault tolerance without durable storage");
    }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Raise maxConcurrentStages to at least 2 in the query context (default is 2).
2. If a single concurrent stage is genuinely desired, disable pipelining instead.
3. If the value comes from user input, validate/adjust it before constructing the kernel: if pipeline && maxConcurrentStages < 2, set it to 2.

Example fix

// before
context.put("maxConcurrentStages", 1);
context.put("pipelinedStagesEnabled", true);

// after
context.put("maxConcurrentStages", 2);
context.put("pipelinedStagesEnabled", true);

if (pipeline && maxConcurrentStages < 2) {
  maxConcurrentStages = 2; // pipelining requires >= 2
}

try {
  kernelConfig = buildKernelConfig(ctx);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains(">= 2 when pipelining")) {
    ctx.put("maxConcurrentStages", 2);
    kernelConfig = buildKernelConfig(ctx);
  }
}

When it happens

Trigger: Constructing ControllerQueryKernelConfig with pipeline=true and maxConcurrentStages set to 0 or 1, typically via query context keys maxConcurrentStages=1 together with enablePipelinedStages/pipelining enabled.

Common situations: Users manually tuning 'maxConcurrentStagesPerWorker' down to 1 to limit memory while leaving pipelining on; templates copied from non-pipelined setups; automated config that clamps the value to 1 without disabling pipelining.

Related errors

• maxRetainedPartitionSketchBytes must be positive
• maxConcurrentStagesPerWorker must be positive
• Cannot pipeline with fault tolerance
• Cannot pipeline with durable storage
• Cannot have fault tolerance without durable storage

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/562680efa24e69aa. Report an issue: GitHub.