apache/druid · error · IllegalArgumentException

Cannot have fault tolerance without durable storage

Error message

Cannot have fault tolerance without durable storage

What it means

Fault tolerance requires that intermediate results be durably stored so stages can be re-run after worker failure; the constructor throws IAE when faultTolerance is true but durableStorage is false. Durable storage is the enabling prerequisite for fault tolerance in MSQ.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java:82

    if (pipeline && maxConcurrentStages < 2) {
      throw new IAE("maxConcurrentStagesPerWorker must be >= 2 when pipelining");
    }

    if (maxConcurrentStages <= 0) {
      throw new IAE("maxConcurrentStagesPerWorker must be positive");
    }

    if (pipeline && faultTolerance) {
      throw new IAE("Cannot pipeline with fault tolerance");
    }

    if (pipeline && durableStorage) {
      throw new IAE("Cannot pipeline with durable storage");
    }

    if (faultTolerance && !durableStorage) {
      throw new IAE("Cannot have fault tolerance without durable storage");
    }

    this.maxRetainedPartitionSketchBytes = maxRetainedPartitionSketchBytes;
    this.maxConcurrentStages = maxConcurrentStages;
    this.pipeline = pipeline;
    this.durableStorage = durableStorage;
    this.faultTolerance = faultTolerance;
    this.destination = destination;
    this.controllerHost = controllerHost;
    this.workerIds = workerIds;
    this.workerContextMap = workerContextMap;
  }

  public static Builder builder()
  {
    return new Builder();
  }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Also set durablyStoreTempResults=true (and configure druid.msq.intermediate.storage.* with a valid deep storage location) when enabling fault tolerance.
2. Or disable faultTolerance if durable storage cannot be configured.
3. In query-building code, set faultTolerance implies durableStorage programmatically before constructing the kernel.

Example fix

// before
context.put("faultTolerance", true);

// after
context.put("faultTolerance", true);
context.put("durablyStoreTempResults", true);

if (ctx.getBoolean("faultTolerance") && !ctx.getBoolean("durablyStoreTempResults")) {
  ctx.put("durablyStoreTempResults", true); // fault tolerance implies durable storage
}

try {
  submitQuery(ctx);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("without durable storage")) {
    ctx.put("durablyStoreTempResults", true);
    submitQuery(ctx);
  }
}

When it happens

Trigger: Constructing ControllerQueryKernelConfig with faultTolerance=true and durableStorage=false, e.g. query context sets faultTolerance=true but durablyStoreTempResults (or the intermediate storage config) is disabled or missing.

Common situations: Users enabling faultTolerance in query context on clusters where deep storage/intermediate durable storage is not configured; forgetting the companion durablyStoreTempResults=true flag; config templates missing the durable storage section.

Related errors

• maxRetainedPartitionSketchBytes must be positive
• maxConcurrentStagesPerWorker must be >= 2 when pipelining
• maxConcurrentStagesPerWorker must be positive
• Cannot pipeline with fault tolerance
• Cannot pipeline with durable storage

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/aba02dd6ef555d1c. Report an issue: GitHub.