{"record":{"id":"aba02dd6ef555d1c","repo":"apache/druid","slug":"cannot-have-fault-tolerance-without-durable-storag","errorCode":null,"errorMessage":"Cannot have fault tolerance without durable storage","messagePattern":"Cannot have fault tolerance without durable storage","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java","lineNumber":82,"sourceCode":"\n    if (pipeline && maxConcurrentStages < 2) {\n      throw new IAE(\"maxConcurrentStagesPerWorker must be >= 2 when pipelining\");\n    }\n\n    if (maxConcurrentStages <= 0) {\n      throw new IAE(\"maxConcurrentStagesPerWorker must be positive\");\n    }\n\n    if (pipeline && faultTolerance) {\n      throw new IAE(\"Cannot pipeline with fault tolerance\");\n    }\n\n    if (pipeline && durableStorage) {\n      throw new IAE(\"Cannot pipeline with durable storage\");\n    }\n\n    if (faultTolerance && !durableStorage) {\n      throw new IAE(\"Cannot have fault tolerance without durable storage\");\n    }\n\n    this.maxRetainedPartitionSketchBytes = maxRetainedPartitionSketchBytes;\n    this.maxConcurrentStages = maxConcurrentStages;\n    this.pipeline = pipeline;\n    this.durableStorage = durableStorage;\n    this.faultTolerance = faultTolerance;\n    this.destination = destination;\n    this.controllerHost = controllerHost;\n    this.workerIds = workerIds;\n    this.workerContextMap = workerContextMap;\n  }\n\n  public static Builder builder()\n  {\n    return new Builder();\n  }\n","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java#L64-L100","documentation":"Fault tolerance requires that intermediate results be durably stored so stages can be re-run after worker failure; the constructor throws IAE when faultTolerance is true but durableStorage is false. Durable storage is the enabling prerequisite for fault tolerance in MSQ.","triggerScenarios":"Constructing ControllerQueryKernelConfig with faultTolerance=true and durableStorage=false, e.g. query context sets faultTolerance=true but durablyStoreTempResults (or the intermediate storage config) is disabled or missing.","commonSituations":"Users enabling faultTolerance in query context on clusters where deep storage/intermediate durable storage is not configured; forgetting the companion durablyStoreTempResults=true flag; config templates missing the durable storage section.","solutions":["Also set durablyStoreTempResults=true (and configure druid.msq.intermediate.storage.* with a valid deep storage location) when enabling fault tolerance.","Or disable faultTolerance if durable storage cannot be configured.","In query-building code, set faultTolerance implies durableStorage programmatically before constructing the kernel."],"exampleFix":"// before\ncontext.put(\"faultTolerance\", true);\n\n// after\ncontext.put(\"faultTolerance\", true);\ncontext.put(\"durablyStoreTempResults\", true);","handlingStrategy":"validation","validationCode":"if (ctx.getBoolean(\"faultTolerance\") && !ctx.getBoolean(\"durablyStoreTempResults\")) {\n  ctx.put(\"durablyStoreTempResults\", true); // fault tolerance implies durable storage\n}","typeGuard":null,"tryCatchPattern":"try {\n  submitQuery(ctx);\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().contains(\"without durable storage\")) {\n    ctx.put(\"durablyStoreTempResults\", true);\n    submitQuery(ctx);\n  }\n}","preventionTips":["Always enable faultTolerance and durablyStoreTempResults together.","Ensure druid.msq.intermediate.storage.* deep-storage settings exist on the cluster.","Add a pre-submit check that derives durableStorage from faultTolerance automatically."],"tags":["msq","configuration","fault-tolerance","durable-storage","validation"],"backgroundTag":"conflicting-config-options","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}