prestodb/presto · error · PrestoException

EXCEEDED_FUNCTION_MEMORY_LIMIT

EXCEEDED_FUNCTION_MEMORY_LIMIT

Error message

The input to %s is too large. More than %s of memory is needed to hold the intermediate hash set.%n

What it means

TypedSet builds an intermediate element Block as a hash set for aggregations like approx_distinct. When the accumulated block exceeds maxBlockMemoryInBytes (derived from MAX_FUNCTION_MEMORY), it throws EXCEEDED_FUNCTION_MEMORY_LIMIT because the aggregation would blow the per-function memory limit.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/TypedSet.java:240

            boolean firstValueNull = elementBlock.isNull(elementBlockPosition);
            Object firstValue = firstValueNull ? defaultValue(elementType.getJavaType()) : readNativeValue(elementType, elementBlock, elementBlockPosition);
            boolean secondValueNull = block.isNull(blockPosition);
            Object secondValue = secondValueNull ? defaultValue(elementType.getJavaType()) : readNativeValue(elementType, block, blockPosition);
            try {
                return !(boolean) elementIsDistinctFrom.get().invoke(firstValue, firstValueNull, secondValue, secondValueNull);
            }
            catch (Throwable t) {
                throw internalError(t);
            }
        }
        return positionEqualsPosition(elementType, elementBlock, elementBlockPosition, block, blockPosition);
    }

    private void addNewElement(int hashPosition, Block block, int position)
    {
        elementType.appendTo(block, position, elementBlock);
        if (elementBlock.getSizeInBytes() - initialElementBlockSizeInBytes > maxBlockMemoryInBytes) {
            throw new PrestoException(
                    EXCEEDED_FUNCTION_MEMORY_LIMIT,
                    format("The input to %s is too large. More than %s of memory is needed to hold the intermediate hash set.%n",
                            functionName,
                            MAX_FUNCTION_MEMORY));
        }
        blockPositionByHash.set(hashPosition, elementBlock.getPositionCount() - 1);

        // increase capacity, if necessary
        size++;
        if (size >= maxFill) {
            rehash();
        }
    }

    private void rehash()
    {
        long newCapacityLong = hashCapacity * 2L;
        if (newCapacityLong > Integer.MAX_VALUE) {

View on GitHub (pinned to 55bb57d202)

Solutions

  1. Increase the query/task function memory limit (memory.max-function-memory query property) if the cluster can afford it.
  2. Reduce input cardinality: pre-aggregate, filter, or hash/bucket the input before the aggregation.
  3. Use a truly approximate sketch (e.g. HLL via approx_distinct's HLL implementation) rather than an exact TypedSet path.
  4. Restrict input width (e.g. truncate long strings) to lower per-element memory.

Example fix

// before
SELECT cardinality(exact_set_agg(user_id)) FROM events; -- billions of distinct ids
// after
SELECT approx_distinct(user_id) FROM events; -- sketch-based, constant memory
Defensive patterns

Strategy: try-catch

Validate before calling

-- estimate distinct count first
SELECT approx_distinct(col) FROM t; -- if in the hundreds of millions, avoid exact set semantics

Try / catch

try { result = query(sql); } catch (PrestoException e) { if (e.getErrorCode().getName().equals("EXCEEDED_FUNCTION_MEMORY_LIMIT")) { result = query(approxSql); } else throw e; }

Prevention

When it happens

Trigger: addNewElement detects elementBlock.getSizeInBytes() - initialElementBlockSizeInBytes > maxBlockMemoryInBytes, i.e. too many distinct high-cardinality values added to the set within one aggregation call.

Common situations: approx_distinct or similar set-based aggregations over columns with billions of distinct values, or wide/varbinary values inflating block size.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/bc388c576766248d. Report an issue: GitHub.