prestodb/presto · error · PrestoException

GENERIC_INSUFFICIENT_RESOURCES

GENERIC_INSUFFICIENT_RESOURCES

Error message

Size of hash table cannot exceed 1 billion entries

What it means

MultiChannelGroupByHash.tryRehash doubles the hash table capacity; if the new capacity would exceed Integer.MAX_VALUE the group-by hash cannot grow further, so it throws GENERIC_INSUFFICIENT_RESOURCES declaring the hard limit of ~1 billion entries. This is a resource/limits error, typically from extremely high group counts.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/operator/MultiChannelGroupByHash.java:385

    {
        if (currentPageBuilder != null) {
            completedPagesMemorySize += currentPageBuilder.getRetainedSizeInBytes();
            currentPageBuilder = currentPageBuilder.newPageBuilderLike();
        }
        else {
            currentPageBuilder = new PageBuilder(types);
        }

        for (int i = 0; i < types.size(); i++) {
            channelBuilders.get(i).add(currentPageBuilder.getBlockBuilder(i));
        }
    }

    private boolean tryRehash()
    {
        long newCapacityLong = hashCapacity * 2L;
        if (newCapacityLong > Integer.MAX_VALUE) {
            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, "Size of hash table cannot exceed 1 billion entries");
        }
        int newCapacity = toIntExact(newCapacityLong);

        // An estimate of how much extra memory is needed before we can go ahead and expand the hash table.
        // This includes the new capacity for groupAddressByHash, rawHashByHashPosition, groupIdsByHash, and groupAddressByGroupId as well as the size of the current page
        preallocatedMemoryInBytes = newCapacity * (long) (Long.BYTES + Integer.BYTES + Byte.BYTES) +
                calculateMaxFill(newCapacity) * Long.BYTES +
                currentPageSizeInBytes;
        if (!updateMemory.update()) {
            // reserved memory but has exceeded the limit
            return false;
        }

        expectedHashCollisions += estimateNumberOfHashCollisions(getGroupCount(), hashCapacity);

        int newMask = newCapacity - 1;
        long[] newKey = new long[newCapacity];
        byte[] rawHashes = new byte[newCapacity];

View on GitHub (pinned to 55bb57d202)

Solutions

1. Reduce cardinality: add coarser GROUP BY keys or pre-aggregate the input
2. Filter out unneeded rows/columns before the aggregation
3. Enable spill / increase memory so lower-cardinality plans execute, or split the query across partitions
4. Raise query limits is not possible for this hard cap; redesign the query instead

Example fix

-- before
SELECT customerId, sessionId, eventTime, count(*) FROM events GROUP BY customerId, sessionId, eventTime;
-- after: bucket to reduce groups
SELECT customerId, date_trunc('hour', eventTime), count(*) FROM events GROUP BY 1, 2;

-- estimate cardinality before running
SELECT approx_distinct(groupKeyExpr) FROM source_table;
-- if result approaches 1e9, restructure the query

try { query(aggSql); } catch (PrestoException e) {
    if (e.getErrorCode().getName().equals("GENERIC_INSUFFICIENT_RESOURCES")) {
        query(aggSqlWithCoarserGrouping); // fallback plan
    } else { throw e; }
}

When it happens

Trigger: tryRehash (called from addNewGroup) when hashCapacity * 2 > Integer.MAX_VALUE, i.e. the group-by hash has grown to ~1 billion (2^30) entries and another group is added.

Common situations: Aggregations with hundreds of millions to billions of distinct grouping keys, runaway DISTINCT values, or a join/aggregate producing a cartesian explosion.

Related errors

• HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT
• GENERIC_INSUFFICIENT_RESOURCES
• GENERIC_INSUFFICIENT_RESOURCES
• MUST_BE_AGGREGATE_OR_GROUP_BY
• CLICKHOUSE_PUSHDOWN_UNSUPPORTED_EXPRESSION

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/95f2aca6e4fd1f82. Report an issue: GitHub.