apache/druid · critical · IAE

druid.processing.buffer.sizeBytes must be less than 2GiB

Error message

druid.processing.buffer.sizeBytes must be less than 2GiB

What it means

Druid's query processing buffer (used for merging query results off-heap) is configured via druid.processing.buffer.sizeBytes. Because the computed buffer size is stored and used as an int, an explicitly configured size larger than Integer.MAX_VALUE (2GiB) is rejected at startup with this IllegalArgumentException. The buffer is allocated as a direct (off-heap) ByteBuffer, so it is intentionally capped below 2GiB.

Solutions

  1. Lower druid.processing.buffer.sizeBytes to at most 2GiB (e.g. 1GiB or 2GiB minus a little).
  2. Remove the explicit setting entirely so Druid uses its default (max of JVM's max direct memory / 10, or the default 1GiB formula).
  3. If more merge memory is needed, increase druid.processing.numMergeBuffers so several sub-2GiB buffers are allocated instead of one huge one.

Example fix

// before
druid.processing.buffer.sizeBytes=3221225472
// after
druid.processing.buffer.sizeBytes=2000000000
Defensive patterns

Strategy: validation

Validate before calling

long size = HumanReadableBytes.parse("3GiB");
if (size > Integer.MAX_VALUE) {
  throw new IllegalArgumentException("druid.processing.buffer.sizeBytes must be < 2GiB, got " + size);
}

Prevention

When it happens

Trigger: Setting druid.processing.buffer.sizeBytes to a value greater than 2^31-1 bytes (e.g. '3GiB' or 3221225472) causes initializeBufferSize to throw when a Druid node (broker/historical) starts and initializes DruidProcessingConfig.

Common situations: Operators bumping the buffer size on memory-rich historicals hoping to speed up large queries; copying settings from machines with huge heap/direct memory; typos like writing sizeBytes in units that end up above 2GiB.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/58efcbad51a42659. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/query/DruidProcessingConfig.java:110

    this.parallelPoolInit = Configs.valueOrDefault(parallelPoolInit, false);

    this.numThreadsConfigured = numThreads != null;
    this.numMergeBuffersConfigured = numMergeBuffers != null;
    initializeBufferSize(runtimeInfo);
  }

  @VisibleForTesting
  public DruidProcessingConfig()
  {
    this(null, null, null, null, null, null, null, null, null, JvmUtils.getRuntimeInfo());
  }

  private void initializeBufferSize(RuntimeInfo runtimeInfo)
  {
    HumanReadableBytes sizeBytesConfigured = this.buffer.getBufferSize();
    if (!DruidProcessingBufferConfig.DEFAULT_PROCESSING_BUFFER_SIZE_BYTES.equals(sizeBytesConfigured)) {
      if (sizeBytesConfigured.getBytes() > Integer.MAX_VALUE) {
        throw new IAE("druid.processing.buffer.sizeBytes must be less than 2GiB");
      }
      computedBufferSizeBytes.set(sizeBytesConfigured.getBytesInInt());
    }

    long directSizeBytes;
    try {
      directSizeBytes = runtimeInfo.getDirectMemorySizeBytes();
      log.info(
          "Detected max direct memory size of [%,d] bytes",
          directSizeBytes
      );
    }
    catch (UnsupportedOperationException e) {
      // max direct memory defaults to max heap size on recent JDK version, unless set explicitly
      directSizeBytes = Runtime.getRuntime().maxMemory() / 4;
      log.info("Using up to [%,d] bytes of direct memory for computation buffers.", directSizeBytes);
    }

View on GitHub (pinned to 9b90983fd2)