prestodb/presto · error · PrestoException

GENERIC_INSUFFICIENT_RESOURCES

GENERIC_INSUFFICIENT_RESOURCES

Error message

Size of hash table cannot exceed 2147483647 entries (%s)

What it means

ValueStore.rehash doubles the internal bucket array of the hash-value store. Bucket count is an int, so when the doubled count would exceed Integer.MAX_VALUE the aggregation throws GENERIC_INSUFFICIENT_RESOURCES, reporting the would-be new capacity. This caps how many distinct hashed values a single ValueStore can hold (~2.1 billion buckets).

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/histogram/ValueStore.java:127

                throw new PrestoException(NOT_SUPPORTED, e.getMessage(), e);
            }
        }
    }

    private int getBucketId(long valueHash, int mask)
    {
        return (int) (valueHash & mask);
    }

    @VisibleForTesting
    void rehash()
    {
        ++rehashCount;

        long newBucketCountLong = bucketCount * 2L;

        if (newBucketCountLong > Integer.MAX_VALUE) {
            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, "Size of hash table cannot exceed " + Integer.MAX_VALUE + " entries (" + newBucketCountLong + ")");
        }

        int newBucketCount = (int) newBucketCountLong;
        int newMask = newBucketCount - 1;

        IntBigArray newBuckets = new IntBigArray(-1);

        newBuckets.ensureCapacity(newBucketCount);

        for (int i = 0; i < values.getPositionCount(); i++) {
            long valueHash = valueHashes.get(i);
            int bucketId = getBucketId(valueHash, newMask);
            int probeCount = 1;

            while (newBuckets.get(bucketId) != EMPTY_BUCKET) {
                int probe = nextProbe(probeCount);

                bucketId = nextBucketId(bucketId, newMask, probe);

View on GitHub (pinned to 55bb57d202)

Solutions

1. Limit the number of distinct values per aggregation instance via filtering, bucketing, or additional GROUP BY keys.
2. Switch to approximate aggregations (approx_distinct, sketches) for very high cardinality.
3. Split the computation into multiple smaller queries/jobs and merge results outside Presto.

Example fix

// before
SELECT histogram(session_id) FROM clicks;
// after
SELECT histogram(session_id) FROM clicks WHERE dt = '2026-09-03'; -- and/or bucket keys first

SELECT approx_distinct(value_col) FROM my_table; -- keep result well below ~2e9

When it happens

Trigger: addAndGetPosition inserting new value hashes into a histogram-style aggregation (ValueStore-backed) until the store's bucketCount * 2L exceeds Integer.MAX_VALUE during rehash.

Common situations: histogram() or similar exact aggregation over billions of distinct values; pathological inputs causing the value store to grow to its hard limit; under-estimated cardinality in test data versus production scale.

Related errors

• GENERIC_INSUFFICIENT_RESOURCES
• GENERIC_INSUFFICIENT_RESOURCES
• EXCEEDED_FUNCTION_MEMORY_LIMIT
• GENERIC_INSUFFICIENT_RESOURCES
• NOT_SUPPORTED

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/ca2b61e2a46c4fb4. Report an issue: GitHub.