{"record":{"id":"bc388c576766248d","repo":"prestodb/presto","slug":"exceeded-function-memory-limit","errorCode":"EXCEEDED_FUNCTION_MEMORY_LIMIT","errorMessage":"The input to %s is too large. More than %s of memory is needed to hold the intermediate hash set.%n","messagePattern":"The input to (.+?) is too large\\. More than (.+?) of memory is needed to hold the intermediate hash set\\.%n","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/TypedSet.java","lineNumber":240,"sourceCode":"            boolean firstValueNull = elementBlock.isNull(elementBlockPosition);\n            Object firstValue = firstValueNull ? defaultValue(elementType.getJavaType()) : readNativeValue(elementType, elementBlock, elementBlockPosition);\n            boolean secondValueNull = block.isNull(blockPosition);\n            Object secondValue = secondValueNull ? defaultValue(elementType.getJavaType()) : readNativeValue(elementType, block, blockPosition);\n            try {\n                return !(boolean) elementIsDistinctFrom.get().invoke(firstValue, firstValueNull, secondValue, secondValueNull);\n            }\n            catch (Throwable t) {\n                throw internalError(t);\n            }\n        }\n        return positionEqualsPosition(elementType, elementBlock, elementBlockPosition, block, blockPosition);\n    }\n\n    private void addNewElement(int hashPosition, Block block, int position)\n    {\n        elementType.appendTo(block, position, elementBlock);\n        if (elementBlock.getSizeInBytes() - initialElementBlockSizeInBytes > maxBlockMemoryInBytes) {\n            throw new PrestoException(\n                    EXCEEDED_FUNCTION_MEMORY_LIMIT,\n                    format(\"The input to %s is too large. More than %s of memory is needed to hold the intermediate hash set.%n\",\n                            functionName,\n                            MAX_FUNCTION_MEMORY));\n        }\n        blockPositionByHash.set(hashPosition, elementBlock.getPositionCount() - 1);\n\n        // increase capacity, if necessary\n        size++;\n        if (size >= maxFill) {\n            rehash();\n        }\n    }\n\n    private void rehash()\n    {\n        long newCapacityLong = hashCapacity * 2L;\n        if (newCapacityLong > Integer.MAX_VALUE) {","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/TypedSet.java#L222-L258","documentation":"TypedSet builds an intermediate element Block as a hash set for aggregations like approx_distinct. When the accumulated block exceeds maxBlockMemoryInBytes (derived from MAX_FUNCTION_MEMORY), it throws EXCEEDED_FUNCTION_MEMORY_LIMIT because the aggregation would blow the per-function memory limit.","triggerScenarios":"addNewElement detects elementBlock.getSizeInBytes() - initialElementBlockSizeInBytes > maxBlockMemoryInBytes, i.e. too many distinct high-cardinality values added to the set within one aggregation call.","commonSituations":"approx_distinct or similar set-based aggregations over columns with billions of distinct values, or wide/varbinary values inflating block size.","solutions":["Increase the query/task function memory limit (memory.max-function-memory query property) if the cluster can afford it.","Reduce input cardinality: pre-aggregate, filter, or hash/bucket the input before the aggregation.","Use a truly approximate sketch (e.g. HLL via approx_distinct's HLL implementation) rather than an exact TypedSet path.","Restrict input width (e.g. truncate long strings) to lower per-element memory."],"exampleFix":"// before\nSELECT cardinality(exact_set_agg(user_id)) FROM events; -- billions of distinct ids\n// after\nSELECT approx_distinct(user_id) FROM events; -- sketch-based, constant memory","handlingStrategy":"try-catch","validationCode":"-- estimate distinct count first\nSELECT approx_distinct(col) FROM t; -- if in the hundreds of millions, avoid exact set semantics","typeGuard":null,"tryCatchPattern":"try { result = query(sql); } catch (PrestoException e) { if (e.getErrorCode().getName().equals(\"EXCEEDED_FUNCTION_MEMORY_LIMIT\")) { result = query(approxSql); } else throw e; }","preventionTips":["Prefer approx_distinct/HLL sketches for high-cardinality columns","Filter or bucket data before exact set aggregations","Tune session memory properties only with capacity planning","Avoid aggregating very wide (varbinary/long string) values in sets"],"tags":["memory-limit","aggregation","high-cardinality"],"backgroundTag":"memory-limit-exceeded","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}