prestodb/presto · critical · PrestoException

GENERIC_INSUFFICIENT_RESOURCES

GENERIC_INSUFFICIENT_RESOURCES

Error message

Size of hash table cannot exceed 1 billion entries

What it means

BigintGroupByHash's open-addressing hash table doubles its capacity on rehash. When the next capacity would exceed Integer.MAX_VALUE (i.e. more than ~1 billion entries), the operator aborts with GENERIC_INSUFFICIENT_RESOURCES, since the group-by hash cannot grow further in memory.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/operator/BigintGroupByHash.java:275

        // record group id in hash
        int groupId = nextGroupId++;

        values.set(hashPosition, value);
        valuesByGroupId.set(groupId, value);
        groupIds.set(hashPosition, groupId);

        // increase capacity, if necessary
        if (needRehash()) {
            tryRehash();
        }
        return groupId;
    }

    private boolean tryRehash()
    {
        long newCapacityLong = hashCapacity * 2L;
        if (newCapacityLong > Integer.MAX_VALUE) {
            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, "Size of hash table cannot exceed 1 billion entries");
        }
        int newCapacity = toIntExact(newCapacityLong);

        // An estimate of how much extra memory is needed before we can go ahead and expand the hash table.
        // This includes the new capacity for values, groupIds, and valuesByGroupId as well as the size of the current page
        preallocatedMemoryInBytes = newCapacity * (long) (Long.BYTES + Integer.BYTES) + calculateMaxFill(newCapacity) * Long.BYTES + currentPageSizeInBytes;
        if (!updateMemory.update()) {
            // reserved memory but has exceeded the limit
            return false;
        }

        expectedHashCollisions += estimateNumberOfHashCollisions(getGroupCount(), hashCapacity);

        int newMask = newCapacity - 1;
        LongBigArray newValues = new LongBigArray();
        newValues.ensureCapacity(newCapacity);
        IntBigArray newGroupIds = new IntBigArray(-1);
        newGroupIds.ensureCapacity(newCapacity);

View on GitHub (pinned to 55bb57d202)

Solutions

Increase parallelism (higher partitioning/node count) so distinct groups are spread across operators
Enable/verify spilling for aggregation (spill-enabled, spill directories) so the hash can evict to disk
Pre-aggregate or reduce cardinality upstream (filter, bucket, or group by a coarser key)
Rewrite the query to aggregate in stages (e.g. approximate via sketches/HyperLogLog)

Example fix

// before
SELECT user_id, count(*) FROM events GROUP BY user_id; -- 2B distinct ids
// after
SELECT approx_distinct(user_id) FROM events; -- or enable spill + increase partitions

Defensive patterns

Strategy: try-catch

Validate before calling

// estimate cardinality before running
// SELECT approx_distinct(group_key) FROM source_table WHERE ...;
// abort or re-plan if the estimate exceeds ~1e9 groups per partition

Try / catch

try {
    result = session.execute(query);
} catch (PrestoException e) {
    if (e.getErrorCode().getName().equals("GENERIC_INSUFFICIENT_RESOURCES")) {
        // re-plan with higher parallelism / enable spill / coarser group key
    } else { throw e; }
}

Prevention

Enable spill (spill-enabled=true with configured spill directories) for high-cardinality aggregations
Partition data so no single operator sees >1B distinct groups
Use approximate aggregation functions for extreme-cardinality analytics
Pre-filter or bucket high-cardinality columns

When it happens

Trigger: GROUP BY / DISTINCT on bigint column(s) where the number of distinct groups exceeds ~1 billion in a single operator instance, causing tryRehash to request double the max capacity.

Common situations: Aggregating on high-cardinality IDs (user IDs, event UUIDs as bigint) over very large scans; insufficient spill config for the group-by hash; under-partitioned data funneling into one driver.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/54b6b2e6f0e5d386. Report an issue: GitHub.