{"record":{"id":"58efcbad51a42659","repo":"apache/druid","slug":"druid-processing-buffer-sizebytes-must-be-less-tha","errorCode":null,"errorMessage":"druid.processing.buffer.sizeBytes must be less than 2GiB","messagePattern":"druid\\.processing\\.buffer\\.sizeBytes must be less than 2GiB","errorType":"validation","errorClass":"IAE","httpStatus":null,"severity":"critical","filePath":"processing/src/main/java/org/apache/druid/query/DruidProcessingConfig.java","lineNumber":110,"sourceCode":"    this.parallelPoolInit = Configs.valueOrDefault(parallelPoolInit, false);\n\n    this.numThreadsConfigured = numThreads != null;\n    this.numMergeBuffersConfigured = numMergeBuffers != null;\n    initializeBufferSize(runtimeInfo);\n  }\n\n  @VisibleForTesting\n  public DruidProcessingConfig()\n  {\n    this(null, null, null, null, null, null, null, null, null, JvmUtils.getRuntimeInfo());\n  }\n\n  private void initializeBufferSize(RuntimeInfo runtimeInfo)\n  {\n    HumanReadableBytes sizeBytesConfigured = this.buffer.getBufferSize();\n    if (!DruidProcessingBufferConfig.DEFAULT_PROCESSING_BUFFER_SIZE_BYTES.equals(sizeBytesConfigured)) {\n      if (sizeBytesConfigured.getBytes() > Integer.MAX_VALUE) {\n        throw new IAE(\"druid.processing.buffer.sizeBytes must be less than 2GiB\");\n      }\n      computedBufferSizeBytes.set(sizeBytesConfigured.getBytesInInt());\n    }\n\n    long directSizeBytes;\n    try {\n      directSizeBytes = runtimeInfo.getDirectMemorySizeBytes();\n      log.info(\n          \"Detected max direct memory size of [%,d] bytes\",\n          directSizeBytes\n      );\n    }\n    catch (UnsupportedOperationException e) {\n      // max direct memory defaults to max heap size on recent JDK version, unless set explicitly\n      directSizeBytes = Runtime.getRuntime().maxMemory() / 4;\n      log.info(\"Using up to [%,d] bytes of direct memory for computation buffers.\", directSizeBytes);\n    }\n","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/query/DruidProcessingConfig.java#L92-L128","documentation":"Druid's query processing buffer (used for merging query results off-heap) is configured via druid.processing.buffer.sizeBytes. Because the computed buffer size is stored and used as an int, an explicitly configured size larger than Integer.MAX_VALUE (2GiB) is rejected at startup with this IllegalArgumentException. The buffer is allocated as a direct (off-heap) ByteBuffer, so it is intentionally capped below 2GiB.","triggerScenarios":"Setting druid.processing.buffer.sizeBytes to a value greater than 2^31-1 bytes (e.g. '3GiB' or 3221225472) causes initializeBufferSize to throw when a Druid node (broker/historical) starts and initializes DruidProcessingConfig.","commonSituations":"Operators bumping the buffer size on memory-rich historicals hoping to speed up large queries; copying settings from machines with huge heap/direct memory; typos like writing sizeBytes in units that end up above 2GiB.","solutions":["Lower druid.processing.buffer.sizeBytes to at most 2GiB (e.g. 1GiB or 2GiB minus a little).","Remove the explicit setting entirely so Druid uses its default (max of JVM's max direct memory / 10, or the default 1GiB formula).","If more merge memory is needed, increase druid.processing.numMergeBuffers so several sub-2GiB buffers are allocated instead of one huge one."],"exampleFix":"// before\ndruid.processing.buffer.sizeBytes=3221225472\n// after\ndruid.processing.buffer.sizeBytes=2000000000","handlingStrategy":"validation","validationCode":"long size = HumanReadableBytes.parse(\"3GiB\");\nif (size > Integer.MAX_VALUE) {\n  throw new IllegalArgumentException(\"druid.processing.buffer.sizeBytes must be < 2GiB, got \" + size);\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep druid.processing.buffer.sizeBytes at or below 2GiB.","Rely on Druid's default sizing rather than hardcoding very large buffers.","Scale merge memory by increasing numMergeBuffers, not individual buffer size."],"tags":["configuration","memory","startup","invalid-config-value"],"backgroundTag":"value-out-of-range","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}