apache/druid · error · IllegalArgumentException

Cannot have fault tolerance without durable storage

Error message

Cannot have fault tolerance without durable storage

What it means

Fault tolerance requires that intermediate results be durably stored so stages can be re-run after worker failure; the constructor throws IAE when faultTolerance is true but durableStorage is false. Durable storage is the enabling prerequisite for fault tolerance in MSQ.

Solutions

1. Also set durablyStoreTempResults=true (and configure druid.msq.intermediate.storage.* with a valid deep storage location) when enabling fault tolerance.
2. Or disable faultTolerance if durable storage cannot be configured.
3. In query-building code, set faultTolerance implies durableStorage programmatically before constructing the kernel.

Example fix

// before
context.put("faultTolerance", true);

// after
context.put("faultTolerance", true);
context.put("durablyStoreTempResults", true);

if (ctx.getBoolean("faultTolerance") && !ctx.getBoolean("durablyStoreTempResults")) {
  ctx.put("durablyStoreTempResults", true); // fault tolerance implies durable storage
}

try {
  submitQuery(ctx);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("without durable storage")) {
    ctx.put("durablyStoreTempResults", true);
    submitQuery(ctx);
  }
}

When it happens

Trigger: Constructing ControllerQueryKernelConfig with faultTolerance=true and durableStorage=false, e.g. query context sets faultTolerance=true but durablyStoreTempResults (or the intermediate storage config) is disabled or missing.

Common situations: Users enabling faultTolerance in query context on clusters where deep storage/intermediate durable storage is not configured; forgetting the companion durablyStoreTempResults=true flag; config templates missing the durable storage section.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Cannot mix sortable and unsortable key columns
• Cannot pipeline with durable storage
• Cannot pipeline with fault tolerance
• Invalid bucketByCount
• maxConcurrentStagesPerWorker must be >= 2 when pipelining

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/aba02dd6ef555d1c. Report an issue: GitHub.

Appendix: source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java:82

    if (pipeline && maxConcurrentStages < 2) {
      throw new IAE("maxConcurrentStagesPerWorker must be >= 2 when pipelining");
    }

    if (maxConcurrentStages <= 0) {
      throw new IAE("maxConcurrentStagesPerWorker must be positive");
    }

    if (pipeline && faultTolerance) {
      throw new IAE("Cannot pipeline with fault tolerance");
    }

    if (pipeline && durableStorage) {
      throw new IAE("Cannot pipeline with durable storage");
    }

    if (faultTolerance && !durableStorage) {
      throw new IAE("Cannot have fault tolerance without durable storage");
    }

    this.maxRetainedPartitionSketchBytes = maxRetainedPartitionSketchBytes;
    this.maxConcurrentStages = maxConcurrentStages;
    this.pipeline = pipeline;
    this.durableStorage = durableStorage;
    this.faultTolerance = faultTolerance;
    this.destination = destination;
    this.controllerHost = controllerHost;
    this.workerIds = workerIds;
    this.workerContextMap = workerContextMap;
  }

  public static Builder builder()
  {
    return new Builder();
  }

View on GitHub (pinned to 9b90983fd2)