apache/druid · error · IllegalStateException

Exceeded memory usage when aggregating type

Error message

Exceeded memory usage when aggregating type [%s], size [%s] is larger than max [%s]

What it means

ExpressionLambdaAggregator enforces a per-row memory budget: after evaluating the aggregation lambda, if the estimated size of the accumulated value exceeds maxSizeBytes it throws ISE to protect the JVM from unbounded aggregation state (common with string/array-producing expressions).

Solutions

Increase the aggregator's maxSizeBytes (or query-level maxBytes context parameter)
Rewrite the expression to accumulate smaller values (e.g. cap string lengths, use hashing)
Reduce cardinality of aggregation keys or split the query

Example fix

// before
context: { "maxBytesInProgress": 5242880 } // too small
// after
context: { "maxBytesInProgress": 104857600 } // raised budget

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check rough size of accumulation target
long estimated = 4L * cardinality; // e.g. bytes per accumulated element
if (estimated > maxBytes) {
  throw new IllegalArgumentException("Aggregation may exceed maxBytes=" + maxBytes);
}

Try / catch

try {
  aggregator.aggregate(bindings);
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Exceeded memory usage when aggregating")) {
    log.warn("Aggregation exceeded maxSizeBytes; raising budget or rewriting expression");
  } else {
    throw e;
  }
}

Prevention

Size maxSizeBytes (and query maxBytes context) for expected cardinality of accumulated values
Cap lengths of strings/arrays produced by aggregation expressions
Monitor for this ISE in high-cardinality groupBy workloads

When it happens

Trigger: Running an expression aggregator (e.g. expression 'filter' aggregator collecting strings or arrays) whose accumulated value grows beyond the configured maxBytes (druid.query.expression... aggregator maxSizeBytes / query context maxBytesInProgress) during aggregate().

Common situations: GroupBy/expression aggregations collecting large strings or arrays; too-low maxSizeBytes config; high-cardinality aggregation results.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/169fc31b40017806. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/query/aggregation/ExpressionLambdaAggregator.java:64

    this.aggregateNullInputs = thePlan.shouldAggregateNullInputs();
    this.inputColumns = thePlan.getInputs();
    this.maxSizeBytes = maxSizeBytes;
  }

  @Override
  public void aggregate()
  {
    if (!aggregateNullInputs) {
      for (String column : inputColumns) {
        if (bindings.get(column) == null) {
          return;
        }
      }
    }
    final ExprEval<?> eval = lambda.eval(bindings);
    final int estimatedSize = eval.type().getNullableStrategy().estimateSizeBytes(eval.value());
    if (estimatedSize > maxSizeBytes) {
      throw new ISE(
          "Exceeded memory usage when aggregating type [%s], size [%s] is larger than max [%s]",
          eval.type().asTypeString(),
          estimatedSize,
          maxSizeBytes
      );
    }
    bindings.accumulate(eval);
    hasValue = true;
  }

  @Nullable
  @Override
  public Object get()
  {
    return hasValue ? bindings.getAccumulator().value() : null;
  }

  @Override

View on GitHub (pinned to 9b90983fd2)