prestodb/presto · error · PrestoException

GENERIC_INSUFFICIENT_RESOURCES

GENERIC_INSUFFICIENT_RESOURCES

Error message

Size of hash table cannot exceed 1 billion entries

What it means

TypedSet's open-addressing hash table doubles its capacity on load-factor overflow. Capacity is capped because it is stored as an int and addresses must stay positive; doubling past Integer.MAX_VALUE would overflow, so it throws GENERIC_INSUFFICIENT_RESOURCES at ~1 billion entries.

Source

Thrown at presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/TypedSet.java:259

                    EXCEEDED_FUNCTION_MEMORY_LIMIT,
                    format("The input to %s is too large. More than %s of memory is needed to hold the intermediate hash set.%n",
                            functionName,
                            MAX_FUNCTION_MEMORY));
        }
        blockPositionByHash.set(hashPosition, elementBlock.getPositionCount() - 1);

        // increase capacity, if necessary
        size++;
        if (size >= maxFill) {
            rehash();
        }
    }

    private void rehash()
    {
        long newCapacityLong = hashCapacity * 2L;
        if (newCapacityLong > Integer.MAX_VALUE) {
            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, "Size of hash table cannot exceed 1 billion entries");
        }
        int newCapacity = (int) newCapacityLong;

        hashCapacity = newCapacity;
        hashMask = newCapacity - 1;
        maxFill = calculateMaxFill(newCapacity);
        blockPositionByHash.size(newCapacity);
        for (int i = 0; i < newCapacity; i++) {
            blockPositionByHash.set(i, EMPTY_SLOT);
        }

        for (int blockPosition = initialElementBlockOffset; blockPosition < elementBlock.getPositionCount(); blockPosition++) {
            blockPositionByHash.set(getHashPositionOfElement(elementBlock, blockPosition), blockPosition);
        }
    }

    private static int calculateMaxFill(int hashSize)
    {

View on GitHub (pinned to 55bb57d202)

Solutions

Switch to a sketch-based approximate aggregation (approx_distinct) instead of exact set semantics.
Reduce cardinality via filtering, GROUP BY partitioning, or pre-aggregation before the set operation.
Split the workload into smaller queries/segments that each stay under the entry cap.

Example fix

// before
SELECT cardinality(set_agg(user_id)) FROM events; -- >1B distinct
// after
SELECT approx_distinct(user_id) FROM events;

Defensive patterns

Strategy: validation

Validate before calling

-- guard: rough distinct estimate must be far below 1B for exact sets
SELECT approx_distinct(col) FROM t; -- if > ~500M expect rehash failure

Try / catch

try { result = query(exactSetSql); } catch (PrestoException e) { if (e.getErrorCode().getName().equals("GENERIC_INSUFFICIENT_RESOURCES") && e.getMessage().contains("1 billion")) { result = query(approxSql); } else throw e; }

Prevention

Use sketch-based (HLL) aggregations beyond ~100M distinct values
Partition the computation with GROUP BY to shrink per-set cardinality
Monitor table cardinality statistics before planning exact distinct queries
Fallback to approximate results when exactness is not required

When it happens

Trigger: rehash() computes hashCapacity * 2L > Integer.MAX_VALUE, i.e. the set already holds close to 1 billion distinct elements and needs to grow again.

Common situations: Exact distinct aggregations (TypedSet users like approx_distinct's exact mode, set_agg) over extremely high-cardinality columns on very large datasets.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/76b3e865e6dab4aa. Report an issue: GitHub.