apache/druid · critical · MSQException

NotEnoughMemoryFault

Error message

NotEnoughMemoryFault

What it means

WorkerMemoryParameters.createInstance computes how much free memory remains for bundles after reserving per-processor and broadcast-buffer memory; when bundleFreeMemory falls below computeMinimumBundleFreeMemory (the minimum a super sorter needs), it refuses to start the worker and throws MSQException(NotEnoughMemoryFault) with the computed required JVM/task memory. It is a startup-time guard that the worker JVM cannot host the required frames, processors, and super sorter simultaneously.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerMemoryParameters.java:216

  )
  {
    final long bundleMemory = computeBundleMemory(memoryIntrospector.memoryPerTask(), maxConcurrentStages);
    final long processorMemory = computeProcessorMemory(
        computeMaxSimultaneousInputChannelsPerProcessor(inputSlices, broadcastInputNumbers),
        frameSize
    );
    final boolean hasBroadcastInputs = !broadcastInputNumbers.isEmpty();
    final long broadcastBufferMemory =
        hasBroadcastInputs ? computeBroadcastBufferMemoryIncludingOverhead(bundleMemory) : 0;
    final int numProcessingThreads = memoryIntrospector.numProcessingThreads();
    final int maxSimultaneousWorkProcessors = Math.min(numProcessingThreads, computeNumInputPartitions(inputSlices));
    final long bundleFreeMemory =
        bundleMemory - maxSimultaneousWorkProcessors * processorMemory - broadcastBufferMemory;

    final long minimumBundleFreeMemory = computeMinimumBundleFreeMemory(frameSize, numFramesPerOutputChannel);
    if (bundleFreeMemory < minimumBundleFreeMemory) {
      final long requiredTaskMemory = (bundleMemory - bundleFreeMemory + minimumBundleFreeMemory) * maxConcurrentStages;
      throw new MSQException(
          new NotEnoughMemoryFault(
              memoryIntrospector.computeJvmMemoryRequiredForTaskMemory(requiredTaskMemory),
              memoryIntrospector.totalMemoryInJvm(),
              memoryIntrospector.memoryPerTask(),
              memoryIntrospector.numTasksInJvm(),
              memoryIntrospector.numProcessingThreads(),
              computeNumInputWorkers(inputSlices),
              maxConcurrentStages
          )
      );
    }

    // Compute memory breakdown for super-sorting bundles.
    final int partitionStatsMemory =
        StageDefinition.mustGatherResultKeyStatistics(shuffleSpec) ? computePartitionStatsMemory(bundleFreeMemory) : 0;
    final long superSorterMemory = bundleFreeMemory - partitionStatsMemory;
    final int maxOutputPartitions = computeMaxOutputPartitions(shuffleSpec);

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Increase worker task memory (druid.indexer.runner/javaOpts or the task context's druid.msq.task.memory adjustments) so requiredTaskMemory fits in the JVM
2. Reduce druid.processing.numThreads on the worker so fewer processor reservations are made
3. Lower MSQ frame size / numFramesPerOutputChannel / maxConcurrentStages context settings
4. Read NotEnoughMemoryFault.suggestedServerMemory from the fault and size the worker to at least that value

Example fix

// before: worker with 1GB heap and 8 processing threads, MSQ fails to start
// after: raise heap and cap threads
// runtime.properties / task javaOpts: -Xmx4g
// druid.processing.numThreads=2
// context: {"maxConcurrentStages": 1}

// ensure worker heap comfortably exceeds requirement before submitting
long heapBytes = Runtime.getRuntime().maxMemory();
long required = threads * processorMem + broadcastMem + minBundleFree;
if (heapBytes < required) throw new IllegalStateException("Increase worker -Xmx to >= " + required);

try {
  startWorker();
} catch (MSQException e) {
  if (e.getFault() instanceof NotEnoughMemoryFault f) {
    log.error("Need %d JVM bytes, have %d", f.getRequiredMemoryInJvm(), f.getTotalMemoryInJvm());
    // resize task heap using f.getSuggestedServerMemory()
  } else { throw e; }
}

When it happens

Trigger: Small druid.processing buffer or too many processing threads relative to -Xmx; large frame size (druid.msq.heap.task.counter or frame config) or many frames per output channel; too many concurrent stages (maxConcurrentStages) multiplying the requirement; high druid.worker.memory memoryPerTask settings.

Common situations: Running MSQ on small peon tasks (e.g. 1GB) with default frame sizes; increasing maxConcurrentStages or processing threads without raising task memory; using large sort/spill configurations on undersized workers; after upgrading Druid to versions that reserve more memory for broadcast buffers.

Related errors

• Invalid value of %s.maxThreads[%d]
• No such shuffleKind[%s]
• maxRetainedPartitionSketchBytes must be positive
• maxConcurrentStagesPerWorker must be >= 2 when pipelining
• maxConcurrentStagesPerWorker must be positive

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/f8e045b988efa6d6. Report an issue: GitHub.