{"record":{"id":"169fc31b40017806","repo":"apache/druid","slug":"exceeded-memory-usage-when-aggregating-type-s","errorCode":null,"errorMessage":"Exceeded memory usage when aggregating type [%s], size [%s] is larger than max [%s]","messagePattern":"Exceeded memory usage when aggregating type \\[(.+?)\\], size \\[(.+?)\\] is larger than max \\[(.+?)\\]","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/query/aggregation/ExpressionLambdaAggregator.java","lineNumber":64,"sourceCode":"    this.aggregateNullInputs = thePlan.shouldAggregateNullInputs();\n    this.inputColumns = thePlan.getInputs();\n    this.maxSizeBytes = maxSizeBytes;\n  }\n\n  @Override\n  public void aggregate()\n  {\n    if (!aggregateNullInputs) {\n      for (String column : inputColumns) {\n        if (bindings.get(column) == null) {\n          return;\n        }\n      }\n    }\n    final ExprEval<?> eval = lambda.eval(bindings);\n    final int estimatedSize = eval.type().getNullableStrategy().estimateSizeBytes(eval.value());\n    if (estimatedSize > maxSizeBytes) {\n      throw new ISE(\n          \"Exceeded memory usage when aggregating type [%s], size [%s] is larger than max [%s]\",\n          eval.type().asTypeString(),\n          estimatedSize,\n          maxSizeBytes\n      );\n    }\n    bindings.accumulate(eval);\n    hasValue = true;\n  }\n\n  @Nullable\n  @Override\n  public Object get()\n  {\n    return hasValue ? bindings.getAccumulator().value() : null;\n  }\n\n  @Override","sourceCodeStart":46,"sourceCodeEnd":82,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/query/aggregation/ExpressionLambdaAggregator.java#L46-L82","documentation":"ExpressionLambdaAggregator enforces a per-row memory budget: after evaluating the aggregation lambda, if the estimated size of the accumulated value exceeds maxSizeBytes it throws ISE to protect the JVM from unbounded aggregation state (common with string/array-producing expressions).","triggerScenarios":"Running an expression aggregator (e.g. expression 'filter' aggregator collecting strings or arrays) whose accumulated value grows beyond the configured maxBytes (druid.query.expression... aggregator maxSizeBytes / query context maxBytesInProgress) during aggregate().","commonSituations":"GroupBy/expression aggregations collecting large strings or arrays; too-low maxSizeBytes config; high-cardinality aggregation results.","solutions":["Increase the aggregator's maxSizeBytes (or query-level maxBytes context parameter)","Rewrite the expression to accumulate smaller values (e.g. cap string lengths, use hashing)","Reduce cardinality of aggregation keys or split the query"],"exampleFix":"// before\ncontext: { \"maxBytesInProgress\": 5242880 } // too small\n// after\ncontext: { \"maxBytesInProgress\": 104857600 } // raised budget","handlingStrategy":"try-catch","validationCode":"// pre-check rough size of accumulation target\nlong estimated = 4L * cardinality; // e.g. bytes per accumulated element\nif (estimated > maxBytes) {\n  throw new IllegalArgumentException(\"Aggregation may exceed maxBytes=\" + maxBytes);\n}","typeGuard":null,"tryCatchPattern":"try {\n  aggregator.aggregate(bindings);\n} catch (IllegalStateException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Exceeded memory usage when aggregating\")) {\n    log.warn(\"Aggregation exceeded maxSizeBytes; raising budget or rewriting expression\");\n  } else {\n    throw e;\n  }\n}","preventionTips":["Size maxSizeBytes (and query maxBytes context) for expected cardinality of accumulated values","Cap lengths of strings/arrays produced by aggregation expressions","Monitor for this ISE in high-cardinality groupBy workloads"],"tags":["aggregation","memory-limit","expression"],"backgroundTag":"payload-too-large","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}