apache/druid · critical · ProvisionException

Not enough direct memory. Please adjust…

Error message

Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numMergeBuffers: maxDirectMemory[%,d], memoryNeeded[%,d] = druid.processing.buffer.sizeBytes[%,d] * (druid.processing.numMergeBuffers[%,d] + 1)

What it means

BrokerProcessingModule.verifyDirectMemory checks at Guice provision time that the JVM's max direct memory can hold the intermediate-results pool plus all merge buffers: buffer.sizeBytes * (numMergeBuffers + 1). If MaxDirectMemorySize is smaller, provisioning fails with this ProvisionException so the broker refuses to start misconfigured.

Solutions

Increase -XX:MaxDirectMemorySize to at least buffer.sizeBytes * (numMergeBuffers + 1)
Lower druid.processing.buffer.sizeBytes
Lower druid.processing.numMergeBuffers
Verify with the printed maxDirectMemory/memoryNeeded numbers in the message

Example fix

// before
JAVA_OPTS="-Xmx4g" # direct memory defaults too low
// after
JAVA_OPTS="-Xmx4g -XX:MaxDirectMemorySize=6g"

Defensive patterns

Strategy: validation

Validate before calling

long bufferBytes = config.intermediateComputeSizeBytes();
long needed = bufferBytes * (config.getNumMergeBuffers() + 1L);
if (runtimeInfo.getDirectMemorySizeBytes() < needed) {
  throw new IllegalStateException("raise -XX:MaxDirectMemorySize to >= " + needed);
}

Try / catch

try {
  injector = GuiceInjectors.makeStartupInjector(...);
} catch (ProvisionException e) {
  if (e.getMessage().contains("Not enough direct memory")) {
    log.error("Reconfigure -XX:MaxDirectMemorySize or druid.processing buffers");
    System.exit(1);
  }
  throw e;
}

Prevention

Set -XX:MaxDirectMemorySize explicitly in every broker start script/container
Recompute direct memory whenever numMergeBuffers or buffer.sizeBytes changes
In containers, account for JVM direct memory beyond heap limits
Smoke-test broker startup after processing config changes

When it happens

Trigger: Starting a Broker where druid.processing.buffer.sizeBytes * (druid.processing.numMergeBuffers + 1) exceeds -XX:MaxDirectMemorySize; getIntermediateResultsPool or getMergeBufferPool is provisioned.

Common situations: Docker/K8s containers with low default direct memory; raising numMergeBuffers or buffer size without raising -XX:MaxDirectMemorySize; forgetting that JVM default max direct memory may equal heap size.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/af372c0935dfaa2a. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/guice/BrokerProcessingModule.java:157

  }

  @Provides
  @Merging
  public ForkJoinPool getMergeProcessingPool(LifecycleForkJoinPoolProvider poolProvider)
  {
    return poolProvider.getPool();
  }

  private void verifyDirectMemory(DruidProcessingConfig config, RuntimeInfo runtimeInfo)
  {
    final long memoryNeeded = (long) config.intermediateComputeSizeBytes() *
                              (config.getNumMergeBuffers() + 1);

    try {
      final long maxDirectMemory = runtimeInfo.getDirectMemorySizeBytes();

      if (maxDirectMemory < memoryNeeded) {
        throw new ProvisionException(
            StringUtils.format(
                "Not enough direct memory.  Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numMergeBuffers: "
                + "maxDirectMemory[%,d], memoryNeeded[%,d] = druid.processing.buffer.sizeBytes[%,d] * (druid.processing.numMergeBuffers[%,d] + 1)",
                maxDirectMemory,
                memoryNeeded,
                config.intermediateComputeSizeBytes(),
                config.getNumMergeBuffers()
            )
        );
      }
    }
    catch (UnsupportedOperationException e) {
      log.debug("Checking for direct memory size is not support on this platform: %s", e);
      log.info(
          "Your memory settings require at least %,d bytes of direct memory. "
          + "Your machine must have at least this much memory available, and your JVM "
          + "-XX:MaxDirectMemorySize parameter must be at least this high. "
          + "If it is, you may safely ignore this message. "

View on GitHub (pinned to 9b90983fd2)