apache/druid · error · IllegalArgumentException

maxRetainedPartitionSketchBytes must be positive

Error message

maxRetainedPartitionSketchBytes must be positive

What it means

ControllerQueryKernelConfig's constructor validates maxRetainedPartitionSketchBytes and throws IAE when it is <= 0. This setting bounds how many bytes of partition-boundary sketch data the controller retains per partition during shuffle; a non-positive value is meaningless and rejected eagerly at kernel construction.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/kernel/controller/ControllerQueryKernelConfig.java:62

  private final String controllerHost;
  @Nullable
  private final List<String> workerIds;
  private final Map<String, Object> workerContextMap;

  ControllerQueryKernelConfig(
      int maxRetainedPartitionSketchBytes,
      int maxConcurrentStages,
      boolean pipeline,
      boolean durableStorage,
      boolean faultTolerance,
      MSQDestination destination,
      @Nullable String controllerHost,
      @Nullable List<String> workerIds,
      Map<String, Object> workerContextMap
  )
  {
    if (maxRetainedPartitionSketchBytes <= 0) {
      throw new IAE("maxRetainedPartitionSketchBytes must be positive");
    }

    if (pipeline && maxConcurrentStages < 2) {
      throw new IAE("maxConcurrentStagesPerWorker must be >= 2 when pipelining");
    }

    if (maxConcurrentStages <= 0) {
      throw new IAE("maxConcurrentStagesPerWorker must be positive");
    }

    if (pipeline && faultTolerance) {
      throw new IAE("Cannot pipeline with fault tolerance");
    }

    if (pipeline && durableStorage) {
      throw new IAE("Cannot pipeline with durable storage");
    }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Remove the override so the built-in default (1073741824, 1 GiB) applies.
2. Set maxRetainedPartitionSketchBytes in the query context to a positive integer, e.g. "maxRetainedPartitionSketchBytes": 1073741824.
3. If computed dynamically, clamp it: Math.max(1, computedValue) before constructing the config.
4. Check for string-to-int parsing of an empty/unset value yielding 0 in the caller that builds the config.

Example fix

// before
context.put("maxRetainedPartitionSketchBytes", 0);

// after
context.put("maxRetainedPartitionSketchBytes", 1073741824); // 1 GiB

final int sketchBytes = context.getInt("maxRetainedPartitionSketchBytes");
if (sketchBytes <= 0) {
  throw new IllegalArgumentException("maxRetainedPartitionSketchBytes must be > 0, got " + sketchBytes);
}

try {
  kernelConfig = ControllerQueryKernelConfig.builder()...build();
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("maxRetainedPartitionSketchBytes")) {
    // fall back to default 1073741824 and rebuild
  }
}

When it happens

Trigger: Building a ControllerQueryKernel (or the enclosing controller config) with maxRetainedPartitionSketchBytes set to 0 or a negative number, e.g. from a context key like maxRetainedPartitionSketchBytes bound to 0, an unset/empty numeric parsed as 0, or a negative value supplied in the query context.

Common situations: Users setting druid.msq.maxRetainedPartitionSketchBytes (or the query context equivalent) to 0 expecting 'unlimited', typos producing negative values, or programmatic config builders defaulting the int to 0 instead of the proper default (e.g. 1GB, 1073741824).

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• maxConcurrentStagesPerWorker must be positive
• maxConcurrentStagesPerWorker must be >= 2 when pipelining
• Cannot have fault tolerance without durable storage
• Invalid bucketByCount [%d]
• Cannot mix sortable and unsortable key columns

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/329a7964689654c8. Report an issue: GitHub.