{"record":{"id":"76b3e865e6dab4aa","repo":"prestodb/presto","slug":"generic-insufficient-resources-76b3e8","errorCode":"GENERIC_INSUFFICIENT_RESOURCES","errorMessage":"Size of hash table cannot exceed 1 billion entries","messagePattern":"Size of hash table cannot exceed 1 billion entries","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/TypedSet.java","lineNumber":259,"sourceCode":"                    EXCEEDED_FUNCTION_MEMORY_LIMIT,\n                    format(\"The input to %s is too large. More than %s of memory is needed to hold the intermediate hash set.%n\",\n                            functionName,\n                            MAX_FUNCTION_MEMORY));\n        }\n        blockPositionByHash.set(hashPosition, elementBlock.getPositionCount() - 1);\n\n        // increase capacity, if necessary\n        size++;\n        if (size >= maxFill) {\n            rehash();\n        }\n    }\n\n    private void rehash()\n    {\n        long newCapacityLong = hashCapacity * 2L;\n        if (newCapacityLong > Integer.MAX_VALUE) {\n            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, \"Size of hash table cannot exceed 1 billion entries\");\n        }\n        int newCapacity = (int) newCapacityLong;\n\n        hashCapacity = newCapacity;\n        hashMask = newCapacity - 1;\n        maxFill = calculateMaxFill(newCapacity);\n        blockPositionByHash.size(newCapacity);\n        for (int i = 0; i < newCapacity; i++) {\n            blockPositionByHash.set(i, EMPTY_SLOT);\n        }\n\n        for (int blockPosition = initialElementBlockOffset; blockPosition < elementBlock.getPositionCount(); blockPosition++) {\n            blockPositionByHash.set(getHashPositionOfElement(elementBlock, blockPosition), blockPosition);\n        }\n    }\n\n    private static int calculateMaxFill(int hashSize)\n    {","sourceCodeStart":241,"sourceCodeEnd":277,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/TypedSet.java#L241-L277","documentation":"TypedSet's open-addressing hash table doubles its capacity on load-factor overflow. Capacity is capped because it is stored as an int and addresses must stay positive; doubling past Integer.MAX_VALUE would overflow, so it throws GENERIC_INSUFFICIENT_RESOURCES at ~1 billion entries.","triggerScenarios":"rehash() computes hashCapacity * 2L > Integer.MAX_VALUE, i.e. the set already holds close to 1 billion distinct elements and needs to grow again.","commonSituations":"Exact distinct aggregations (TypedSet users like approx_distinct's exact mode, set_agg) over extremely high-cardinality columns on very large datasets.","solutions":["Switch to a sketch-based approximate aggregation (approx_distinct) instead of exact set semantics.","Reduce cardinality via filtering, GROUP BY partitioning, or pre-aggregation before the set operation.","Split the workload into smaller queries/segments that each stay under the entry cap."],"exampleFix":"// before\nSELECT cardinality(set_agg(user_id)) FROM events; -- >1B distinct\n// after\nSELECT approx_distinct(user_id) FROM events;","handlingStrategy":"validation","validationCode":"-- guard: rough distinct estimate must be far below 1B for exact sets\nSELECT approx_distinct(col) FROM t; -- if > ~500M expect rehash failure","typeGuard":null,"tryCatchPattern":"try { result = query(exactSetSql); } catch (PrestoException e) { if (e.getErrorCode().getName().equals(\"GENERIC_INSUFFICIENT_RESOURCES\") && e.getMessage().contains(\"1 billion\")) { result = query(approxSql); } else throw e; }","preventionTips":["Use sketch-based (HLL) aggregations beyond ~100M distinct values","Partition the computation with GROUP BY to shrink per-set cardinality","Monitor table cardinality statistics before planning exact distinct queries","Fallback to approximate results when exactness is not required"],"tags":["memory-limit","hash-table","high-cardinality"],"backgroundTag":"insufficient-resources","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}