apache/druid · error · ResourceLimitExceededException

Query needs merge buffers, but only merge buffers were…

Error message

Query needs %s merge buffers, but only %s merge buffers were configured

What it means

Before running a group-by query, GroupingEngine.prepareResource computes the number of merge buffers needed for toolchest merge plus the merging query runner, and throws ResourceLimitExceededException when this exceeds the configured merge buffer pool size (druid.processing.numMergeBuffers).

Solutions

1. Increase druid.processing.numMergeBuffers on historical/broker nodes
2. Reduce subquery nesting depth or rewrite the query to fewer levels
3. Lower concurrency (druid.processing.numThreads) so fewer queries contend, or route heavy nested queries to dedicated tier

Example fix

// before (runtime.properties)
druid.processing.numMergeBuffers=2
// after
druid.processing.numMergeBuffers=4

if (requiredMergeBuffers(query) > mergeBufferPool.maxSize()) { throw new ResourceLimitExceededException("Reduce nesting or increase druid.processing.numMergeBuffers"); }

null

try { groupingEngine.prepareResource(query, ...); } catch (ResourceLimitExceededException e) { /* surface a 429-style error or retry on a larger cluster */ }

When it happens

Trigger: Submitting a group-by query requiring more merge buffers than available: multi-level groupBy with multiple nested subqueries, high druid.query.groupBy.maxMergingDictionarySize configurations aside — specifically nested subquery depth or broker merge requirements pushing requiredMergeBufferNum above pool maxSize.

Common situations: Cluster configured with default numMergeBuffers while running deeply nested groupBy subqueries; many concurrent groupBy queries exhausting (or exceeding sizing for) merge buffers; v2 engine setups where subquery nesting multiplies buffer requirements.

Related errors

• Ambiguous build, limit
• Ambiguous build, limitSpec
• buffer for list is too small, was
• Cannot force limit push down when a having spec is present.
• Duplicate output name

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/d356f47c04da8ffa. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/query/groupby/GroupingEngine.java:174

      BlockingPool<ByteBuffer> mergeBufferPool,
      boolean usesGroupByMergingQueryRunner,
      GroupByQueryConfig groupByQueryConfig
  )
  {

    final int requiredMergeBufferNumForToolchestMerge =
        GroupByQueryResources.countRequiredMergeBufferNumForToolchestMerge(query);

    final int requiredMergeBufferNumForMergingQueryRunner =
        usesGroupByMergingQueryRunner
        ? GroupByQueryResources.countRequiredMergeBufferNumForMergingQueryRunner(groupByQueryConfig, query)
        : 0;

    final int requiredMergeBufferNum =
        requiredMergeBufferNumForToolchestMerge + requiredMergeBufferNumForMergingQueryRunner;

    if (requiredMergeBufferNum > mergeBufferPool.maxSize()) {
      throw new ResourceLimitExceededException(
          "Query needs " + requiredMergeBufferNum + " merge buffers, but only "
          + mergeBufferPool.maxSize() + " merge buffers were configured"
      );
    } else if (requiredMergeBufferNum == 0) {
      return new GroupByQueryResources(null, null);
    } else {
      final List<ReferenceCountingResourceHolder<ByteBuffer>> mergeBufferHolders;
      final QueryContext context = query.context();
      if (context.hasTimeout()) {
        mergeBufferHolders = mergeBufferPool.takeBatch(requiredMergeBufferNum, context.getTimeout());
      } else {
        mergeBufferHolders = mergeBufferPool.takeBatch(requiredMergeBufferNum);
      }
      if (mergeBufferHolders.isEmpty()) {
        throw QueryCapacityExceededException.withErrorMessageAndResolvedHost(
            StringUtils.format(
                "Cannot acquire %s merge buffers. Try again after current running queries are finished.",
                requiredMergeBufferNum

View on GitHub (pinned to 9b90983fd2)