{"record":{"id":"562680efa24e69aa","repo":"apache/druid","slug":"maxconcurrentstagesperworker-must-be-2-when-pip","errorCode":null,"errorMessage":"maxConcurrentStagesPerWorker must be >= 2 when pipelining","messagePattern":"maxConcurrentStagesPerWorker must be >= 2 when pipelining","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java","lineNumber":66,"sourceCode":"\n  ControllerQueryKernelConfig(\n      int maxRetainedPartitionSketchBytes,\n      int maxConcurrentStages,\n      boolean pipeline,\n      boolean durableStorage,\n      boolean faultTolerance,\n      MSQDestination destination,\n      @Nullable String controllerHost,\n      @Nullable List<String> workerIds,\n      Map<String, Object> workerContextMap\n  )\n  {\n    if (maxRetainedPartitionSketchBytes <= 0) {\n      throw new IAE(\"maxRetainedPartitionSketchBytes must be positive\");\n    }\n\n    if (pipeline && maxConcurrentStages < 2) {\n      throw new IAE(\"maxConcurrentStagesPerWorker must be >= 2 when pipelining\");\n    }\n\n    if (maxConcurrentStages <= 0) {\n      throw new IAE(\"maxConcurrentStagesPerWorker must be positive\");\n    }\n\n    if (pipeline && faultTolerance) {\n      throw new IAE(\"Cannot pipeline with fault tolerance\");\n    }\n\n    if (pipeline && durableStorage) {\n      throw new IAE(\"Cannot pipeline with durable storage\");\n    }\n\n    if (faultTolerance && !durableStorage) {\n      throw new IAE(\"Cannot have fault tolerance without durable storage\");\n    }\n","sourceCodeStart":48,"sourceCodeEnd":84,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java#L48-L84","documentation":"When pipelining is enabled, the kernel must run at least two stages concurrently per worker, so the constructor rejects maxConcurrentStages < 2 with IAE. Pipelining with only one concurrent stage slot is contradictory and cannot be scheduled.","triggerScenarios":"Constructing ControllerQueryKernelConfig with pipeline=true and maxConcurrentStages set to 0 or 1, typically via query context keys maxConcurrentStages=1 together with enablePipelinedStages/pipelining enabled.","commonSituations":"Users manually tuning 'maxConcurrentStagesPerWorker' down to 1 to limit memory while leaving pipelining on; templates copied from non-pipelined setups; automated config that clamps the value to 1 without disabling pipelining.","solutions":["Raise maxConcurrentStages to at least 2 in the query context (default is 2).","If a single concurrent stage is genuinely desired, disable pipelining instead.","If the value comes from user input, validate/adjust it before constructing the kernel: if pipeline && maxConcurrentStages < 2, set it to 2."],"exampleFix":"// before\ncontext.put(\"maxConcurrentStages\", 1);\ncontext.put(\"pipelinedStagesEnabled\", true);\n\n// after\ncontext.put(\"maxConcurrentStages\", 2);\ncontext.put(\"pipelinedStagesEnabled\", true);","handlingStrategy":"validation","validationCode":"if (pipeline && maxConcurrentStages < 2) {\n  maxConcurrentStages = 2; // pipelining requires >= 2\n}","typeGuard":null,"tryCatchPattern":"try {\n  kernelConfig = buildKernelConfig(ctx);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\">= 2 when pipelining\")) {\n    ctx.put(\"maxConcurrentStages\", 2);\n    kernelConfig = buildKernelConfig(ctx);\n  }\n}","preventionTips":["Keep maxConcurrentStages at its default of 2 unless profiling shows otherwise.","If limiting concurrency, disable pipelining rather than setting the value to 1.","Validate pipelining-related context keys together as a group."],"tags":["msq","configuration","pipelining","validation"],"backgroundTag":"conflicting-config-options","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}