prestodb/presto · error · PrestoException

GENERIC_INSUFFICIENT_RESOURCES

GENERIC_INSUFFICIENT_RESOURCES

Error message

Size of hash table cannot exceed 2147483647 entries (%s)

What it means

GroupedTypedHistogram implements the histogram aggregation's key set as an open-addressing hash table backed by int bucket arrays. When the table doubles on rehash, the new bucket count must fit in a signed int; exceeding Integer.MAX_VALUE entries cannot be represented, so Presto throws GENERIC_INSUFFICIENT_RESOURCES instead of overflowing.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/histogram/GroupedTypedHistogram.java:288

    private void iterateGroupNodes(long groupId, NodeReader nodeReader)
    {
        // while the index can be a long, the value is always an int
        int currentPointer = (int) headPointers.get(groupId);
        checkArgument(currentPointer != NULL, "valid group must have non-null head pointer");

        while (currentPointer != NULL) {
            checkState(currentPointer < nextNodePointer, "error, corrupt pointer; max valid %s, found %s", nextNodePointer, currentPointer);
            nodeReader.read(currentPointer);
            currentPointer = nextPointers.get(currentPointer);
        }
    }

    private void rehash()
    {
        long newBucketCountLong = bucketCount * 2L;

        if (newBucketCountLong > Integer.MAX_VALUE) {
            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, "Size of hash table cannot exceed " + Integer.MAX_VALUE + " entries (" + newBucketCountLong + ")");
        }

        int newBucketCount = computeBucketCount((int) newBucketCountLong, MAX_FILL_RATIO);
        int newMask = newBucketCount - 1;
        IntBigArray newBuckets = new IntBigArray(-1);
        newBuckets.ensureCapacity(newBucketCount);

        for (int i = 0; i < nextNodePointer; i++) {
            // find the old one
            int bucketId = getBucketIdForNode(i, newMask);
            int probeCount = 1;

            int originalBucket = bucketId;
            // find new one
            while (newBuckets.get(bucketId) != -1) {
                int probe = nextProbe(probeCount);
                bucketId = nextBucketId(originalBucket, newMask, probe);
                probeCount++;

View on GitHub (pinned to 55bb57d202)

Solutions

  1. Reduce input cardinality: group more finely, or bucket/round keys before histogram(x).
  2. Use a different summary (approx_distinct, TopN via histogram over cast keys) for very high cardinality.
  3. Pre-aggregate upstream so a single histogram never sees billions of distinct keys.

Example fix

// before
SELECT histogram(user_id) FROM events; -- billions of distinct ids
// after
SELECT histogram(date_trunc('hour', ts)) FROM events; -- or GROUP BY region first
Defensive patterns

Strategy: validation

Validate before calling

-- bound key cardinality before histogram
SELECT approx_distinct(key) FROM t GROUP BY group_key ORDER BY 1 DESC LIMIT 1; -- if near billions, do not use histogram

Try / catch

try { result = query(...); } catch (PrestoException e) { if (e.getErrorCode().getName().equals("GENERIC_INSUFFICIENT_RESOURCES")) { /* fall back to lower-cardinality keys */ } else { throw e; } }

Prevention

When it happens

Trigger: A histogram(x) aggregation accumulating so many distinct key values that the internal hash table's bucket count doubled beyond Integer.MAX_VALUE — i.e. billions of distinct keys in one group.

Common situations: Histogramming a high-cardinality column (ids, timestamps, free text) in a single group; a forgotten GROUP BY so all rows land in one histogram; runaway distinct values from bad joins or data corruption.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/53e6185e51961367. Report an issue: GitHub.