{"record":{"id":"1091337b2d8bd8e6","repo":"apache/druid","slug":"cannot-pipeline-with-fault-tolerance","errorCode":null,"errorMessage":"Cannot pipeline with fault tolerance","messagePattern":"Cannot pipeline with fault tolerance","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java","lineNumber":74,"sourceCode":"      @Nullable String controllerHost,\n      @Nullable List<String> workerIds,\n      Map<String, Object> workerContextMap\n  )\n  {\n    if (maxRetainedPartitionSketchBytes <= 0) {\n      throw new IAE(\"maxRetainedPartitionSketchBytes must be positive\");\n    }\n\n    if (pipeline && maxConcurrentStages < 2) {\n      throw new IAE(\"maxConcurrentStagesPerWorker must be >= 2 when pipelining\");\n    }\n\n    if (maxConcurrentStages <= 0) {\n      throw new IAE(\"maxConcurrentStagesPerWorker must be positive\");\n    }\n\n    if (pipeline && faultTolerance) {\n      throw new IAE(\"Cannot pipeline with fault tolerance\");\n    }\n\n    if (pipeline && durableStorage) {\n      throw new IAE(\"Cannot pipeline with durable storage\");\n    }\n\n    if (faultTolerance && !durableStorage) {\n      throw new IAE(\"Cannot have fault tolerance without durable storage\");\n    }\n\n    this.maxRetainedPartitionSketchBytes = maxRetainedPartitionSketchBytes;\n    this.maxConcurrentStages = maxConcurrentStages;\n    this.pipeline = pipeline;\n    this.durableStorage = durableStorage;\n    this.faultTolerance = faultTolerance;\n    this.destination = destination;\n    this.controllerHost = controllerHost;\n    this.workerIds = workerIds;","sourceCodeStart":56,"sourceCodeEnd":92,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java#L56-L92","documentation":"Pipelining and fault tolerance are mutually exclusive execution modes in the MSQ kernel; the constructor throws IAE when both pipeline and faultTolerance are true. Pipelined stages rely on in-memory exchange while fault tolerance requires checkpointed, replayable stages, so the combination is unsupported.","triggerScenarios":"Constructing ControllerQueryKernelConfig with pipeline=true and faultTolerance=true, e.g. query context having both pipelining enabled and faultTolerance/durablyStoreTempResults-style fault-tolerance flags set.","commonSituations":"Users enabling fault tolerance for long-running ingestion while a cluster default turns on pipelining; combining tuning advice from two different Druid versions where defaults changed.","solutions":["Disable fault tolerance (and its durable-storage prerequisite) to keep pipelining enabled.","Or disable pipelining to keep fault tolerance.","In query-building code, enforce exclusivity before submitting: if (pipeline && faultTolerance) choose one and clear the other context key."],"exampleFix":"// before\ncontext.put(\"pipelinedStagesEnabled\", true);\ncontext.put(\"faultTolerance\", true);\n\n// after\ncontext.put(\"pipelinedStagesEnabled\", true);\ncontext.put(\"faultTolerance\", false); // pick one mode","handlingStrategy":"validation","validationCode":"if (ctx.getBoolean(\"pipelinedStagesEnabled\") && ctx.getBoolean(\"faultTolerance\")) {\n  throw new IllegalArgumentException(\"Enable either pipelining or fault tolerance, not both\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  submitQuery(ctx);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"Cannot pipeline with fault tolerance\")) {\n    ctx.put(\"faultTolerance\", false);\n    submitQuery(ctx);\n  }\n}","preventionTips":["Pick an execution mode (pipelining vs fault tolerance) per query and set only that mode's flags.","Review cluster-default context keys that may implicitly enable fault tolerance.","Document the exclusivity in query templates and internal tooling."],"tags":["msq","configuration","pipelining","fault-tolerance","conflicting-options"],"backgroundTag":"mutually-exclusive-flags","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}