{"record":{"id":"54b6b2e6f0e5d386","repo":"prestodb/presto","slug":"generic-insufficient-resources","errorCode":"GENERIC_INSUFFICIENT_RESOURCES","errorMessage":"Size of hash table cannot exceed 1 billion entries","messagePattern":"Size of hash table cannot exceed 1 billion entries","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"critical","filePath":"presto-main-base/src/main/java/com/facebook/presto/operator/BigintGroupByHash.java","lineNumber":275,"sourceCode":"        // record group id in hash\n        int groupId = nextGroupId++;\n\n        values.set(hashPosition, value);\n        valuesByGroupId.set(groupId, value);\n        groupIds.set(hashPosition, groupId);\n\n        // increase capacity, if necessary\n        if (needRehash()) {\n            tryRehash();\n        }\n        return groupId;\n    }\n\n    private boolean tryRehash()\n    {\n        long newCapacityLong = hashCapacity * 2L;\n        if (newCapacityLong > Integer.MAX_VALUE) {\n            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, \"Size of hash table cannot exceed 1 billion entries\");\n        }\n        int newCapacity = toIntExact(newCapacityLong);\n\n        // An estimate of how much extra memory is needed before we can go ahead and expand the hash table.\n        // This includes the new capacity for values, groupIds, and valuesByGroupId as well as the size of the current page\n        preallocatedMemoryInBytes = newCapacity * (long) (Long.BYTES + Integer.BYTES) + calculateMaxFill(newCapacity) * Long.BYTES + currentPageSizeInBytes;\n        if (!updateMemory.update()) {\n            // reserved memory but has exceeded the limit\n            return false;\n        }\n\n        expectedHashCollisions += estimateNumberOfHashCollisions(getGroupCount(), hashCapacity);\n\n        int newMask = newCapacity - 1;\n        LongBigArray newValues = new LongBigArray();\n        newValues.ensureCapacity(newCapacity);\n        IntBigArray newGroupIds = new IntBigArray(-1);\n        newGroupIds.ensureCapacity(newCapacity);","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-main-base/src/main/java/com/facebook/presto/operator/BigintGroupByHash.java#L257-L293","documentation":"BigintGroupByHash's open-addressing hash table doubles its capacity on rehash. When the next capacity would exceed Integer.MAX_VALUE (i.e. more than ~1 billion entries), the operator aborts with GENERIC_INSUFFICIENT_RESOURCES, since the group-by hash cannot grow further in memory.","triggerScenarios":"GROUP BY / DISTINCT on bigint column(s) where the number of distinct groups exceeds ~1 billion in a single operator instance, causing tryRehash to request double the max capacity.","commonSituations":"Aggregating on high-cardinality IDs (user IDs, event UUIDs as bigint) over very large scans; insufficient spill config for the group-by hash; under-partitioned data funneling into one driver.","solutions":["Increase parallelism (higher partitioning/node count) so distinct groups are spread across operators","Enable/verify spilling for aggregation (spill-enabled, spill directories) so the hash can evict to disk","Pre-aggregate or reduce cardinality upstream (filter, bucket, or group by a coarser key)","Rewrite the query to aggregate in stages (e.g. approximate via sketches/HyperLogLog)"],"exampleFix":"// before\nSELECT user_id, count(*) FROM events GROUP BY user_id; -- 2B distinct ids\n// after\nSELECT approx_distinct(user_id) FROM events; -- or enable spill + increase partitions","handlingStrategy":"try-catch","validationCode":"// estimate cardinality before running\n// SELECT approx_distinct(group_key) FROM source_table WHERE ...;\n// abort or re-plan if the estimate exceeds ~1e9 groups per partition","typeGuard":null,"tryCatchPattern":"try {\n    result = session.execute(query);\n} catch (PrestoException e) {\n    if (e.getErrorCode().getName().equals(\"GENERIC_INSUFFICIENT_RESOURCES\")) {\n        // re-plan with higher parallelism / enable spill / coarser group key\n    } else { throw e; }\n}","preventionTips":["Enable spill (spill-enabled=true with configured spill directories) for high-cardinality aggregations","Partition data so no single operator sees >1B distinct groups","Use approximate aggregation functions for extreme-cardinality analytics","Pre-filter or bucket high-cardinality columns"],"tags":["presto","group-by","hash-table","memory-limit","aggregation"],"backgroundTag":"hash-capacity-exceeded","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}