{"record":{"id":"f8e045b988efa6d6","repo":"apache/druid","slug":"notenoughmemoryfault","errorCode":null,"errorMessage":"NotEnoughMemoryFault","messagePattern":"NotEnoughMemoryFault","errorType":"error_code","errorClass":"MSQException","httpStatus":null,"severity":"critical","filePath":"multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerMemoryParameters.java","lineNumber":216,"sourceCode":"  )\n  {\n    final long bundleMemory = computeBundleMemory(memoryIntrospector.memoryPerTask(), maxConcurrentStages);\n    final long processorMemory = computeProcessorMemory(\n        computeMaxSimultaneousInputChannelsPerProcessor(inputSlices, broadcastInputNumbers),\n        frameSize\n    );\n    final boolean hasBroadcastInputs = !broadcastInputNumbers.isEmpty();\n    final long broadcastBufferMemory =\n        hasBroadcastInputs ? computeBroadcastBufferMemoryIncludingOverhead(bundleMemory) : 0;\n    final int numProcessingThreads = memoryIntrospector.numProcessingThreads();\n    final int maxSimultaneousWorkProcessors = Math.min(numProcessingThreads, computeNumInputPartitions(inputSlices));\n    final long bundleFreeMemory =\n        bundleMemory - maxSimultaneousWorkProcessors * processorMemory - broadcastBufferMemory;\n\n    final long minimumBundleFreeMemory = computeMinimumBundleFreeMemory(frameSize, numFramesPerOutputChannel);\n    if (bundleFreeMemory < minimumBundleFreeMemory) {\n      final long requiredTaskMemory = (bundleMemory - bundleFreeMemory + minimumBundleFreeMemory) * maxConcurrentStages;\n      throw new MSQException(\n          new NotEnoughMemoryFault(\n              memoryIntrospector.computeJvmMemoryRequiredForTaskMemory(requiredTaskMemory),\n              memoryIntrospector.totalMemoryInJvm(),\n              memoryIntrospector.memoryPerTask(),\n              memoryIntrospector.numTasksInJvm(),\n              memoryIntrospector.numProcessingThreads(),\n              computeNumInputWorkers(inputSlices),\n              maxConcurrentStages\n          )\n      );\n    }\n\n    // Compute memory breakdown for super-sorting bundles.\n    final int partitionStatsMemory =\n        StageDefinition.mustGatherResultKeyStatistics(shuffleSpec) ? computePartitionStatsMemory(bundleFreeMemory) : 0;\n    final long superSorterMemory = bundleFreeMemory - partitionStatsMemory;\n    final int maxOutputPartitions = computeMaxOutputPartitions(shuffleSpec);\n","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerMemoryParameters.java#L198-L234","documentation":"WorkerMemoryParameters.createInstance computes how much free memory remains for bundles after reserving per-processor and broadcast-buffer memory; when bundleFreeMemory falls below computeMinimumBundleFreeMemory (the minimum a super sorter needs), it refuses to start the worker and throws MSQException(NotEnoughMemoryFault) with the computed required JVM/task memory. It is a startup-time guard that the worker JVM cannot host the required frames, processors, and super sorter simultaneously.","triggerScenarios":"Small druid.processing buffer or too many processing threads relative to -Xmx; large frame size (druid.msq.heap.task.counter or frame config) or many frames per output channel; too many concurrent stages (maxConcurrentStages) multiplying the requirement; high druid.worker.memory memoryPerTask settings.","commonSituations":"Running MSQ on small peon tasks (e.g. 1GB) with default frame sizes; increasing maxConcurrentStages or processing threads without raising task memory; using large sort/spill configurations on undersized workers; after upgrading Druid to versions that reserve more memory for broadcast buffers.","solutions":["Increase worker task memory (druid.indexer.runner/javaOpts or the task context's druid.msq.task.memory adjustments) so requiredTaskMemory fits in the JVM","Reduce druid.processing.numThreads on the worker so fewer processor reservations are made","Lower MSQ frame size / numFramesPerOutputChannel / maxConcurrentStages context settings","Read NotEnoughMemoryFault.suggestedServerMemory from the fault and size the worker to at least that value"],"exampleFix":"// before: worker with 1GB heap and 8 processing threads, MSQ fails to start\n// after: raise heap and cap threads\n// runtime.properties / task javaOpts: -Xmx4g\n// druid.processing.numThreads=2\n// context: {\"maxConcurrentStages\": 1}","handlingStrategy":"validation","validationCode":"// ensure worker heap comfortably exceeds requirement before submitting\nlong heapBytes = Runtime.getRuntime().maxMemory();\nlong required = threads * processorMem + broadcastMem + minBundleFree;\nif (heapBytes < required) throw new IllegalStateException(\"Increase worker -Xmx to >= \" + required);","typeGuard":null,"tryCatchPattern":"try {\n  startWorker();\n} catch (MSQException e) {\n  if (e.getFault() instanceof NotEnoughMemoryFault f) {\n    log.error(\"Need %d JVM bytes, have %d\", f.getRequiredMemoryInJvm(), f.getTotalMemoryInJvm());\n    // resize task heap using f.getSuggestedServerMemory()\n  } else { throw e; }\n}","preventionTips":["Size worker heaps >= suggestedServerMemory reported by past NotEnoughMemoryFaults","Keep druid.processing.numThreads low relative to heap for MSQ workers","Avoid raising frame size / maxConcurrentStages without scaling memory"],"tags":["msq","memory","configuration"],"backgroundTag":"insufficient-memory","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}