prestodb/presto · error · PrestoException

GENERIC_INSUFFICIENT_RESOURCES

GENERIC_INSUFFICIENT_RESOURCES

Error message

Size of hash table cannot exceed 1 billion entries

What it means

The typed histogram aggregation maintains an open-addressing hash table mapping group IDs to positions. Its capacity is stored as an int, so when doubling during rehash would exceed Integer.MAX_VALUE the aggregation aborts with GENERIC_INSUFFICIENT_RESOURCES, since a histogram with more than ~1 billion distinct entries cannot fit in the in-memory structure. This is a hard resource limit of the implementation, not a query bug.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/histogram/SingleTypedHistogram.java:189

    }

    private void addNewGroup(int hashPosition, int position, Block block, long count)
    {
        hashPositions.set(hashPosition, values.getPositionCount());
        counts.set(values.getPositionCount(), count);
        type.appendTo(block, position, values);

        // increase capacity, if necessary
        if (values.getPositionCount() >= maxFill) {
            rehash();
        }
    }

    private void rehash()
    {
        long newCapacityLong = hashCapacity * 2L;
        if (newCapacityLong > Integer.MAX_VALUE) {
            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, "Size of hash table cannot exceed 1 billion entries");
        }
        int newCapacity = (int) newCapacityLong;

        int newMask = newCapacity - 1;
        IntBigArray newHashPositions = new IntBigArray(-1);
        newHashPositions.ensureCapacity(newCapacity);

        for (int i = 0; i < values.getPositionCount(); i++) {
            // find an empty slot for the address
            int hashPosition = getBucketId(TypeUtils.hashPosition(type, values, i), newMask);

            while (newHashPositions.get(hashPosition) != -1) {
                hashPosition = (hashPosition + 1) & newMask;
            }

            // record the mapping
            newHashPositions.set(hashPosition, i);
        }

View on GitHub (pinned to 55bb57d202)

Solutions

Reduce the number of distinct values fed into histogram() by pre-aggregating, bucketing, or casting keys to a coarser granularity (e.g. truncate timestamps, bucket numeric IDs).
Add WHERE filters or GROUP BY additional columns so each histogram contains fewer distinct values.
If a full histogram is truly needed, extract distinct data in chunks or use approx_distinct / sketches instead of an exact histogram.

Example fix

// before
SELECT histogram(user_id) FROM events;
// after
SELECT histogram(bucketize(user_id)) -- or filter/limit cardinality
SELECT histogram(date_trunc('hour', event_time)) FROM events WHERE dt = CURRENT_DATE;

Defensive patterns

Strategy: validation

Validate before calling

-- Before running: ensure distinct count per histogram group is well under ~1e9
SELECT count(DISTINCT key_col) FROM my_table WHERE <group_filter>;

Prevention

Check cardinality with approx_distinct before using histogram() on large tables.
Bucket or truncate high-cardinality keys before aggregating.
Prefer sketches (approx_distinct, QDigest) over exact histograms at billion-scale.

When it happens

Trigger: Calling the histogram() aggregation (e.g. histogram(col)) on a column whose number of distinct values is so large that SingleTypedHistogram.rehash(), invoked from addNewGroup as new distinct groups are added, would need to double hashCapacity past Integer.MAX_VALUE (capacity * 2L > Integer.MAX_VALUE).

Common situations: Aggregating histogram over a very high-cardinality column (e.g. user IDs, device IDs, request IDs) in a large table; a join or filter bug causing near-unique keys; attempting to build a histogram per group where each group still has billions of distinct values.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/7ae6bbdb8e6d4be0. Report an issue: GitHub.