{"record":{"id":"95f2aca6e4fd1f82","repo":"prestodb/presto","slug":"generic-insufficient-resources-95f2ac","errorCode":"GENERIC_INSUFFICIENT_RESOURCES","errorMessage":"Size of hash table cannot exceed 1 billion entries","messagePattern":"Size of hash table cannot exceed 1 billion entries","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-main-base/src/main/java/com/facebook/presto/operator/MultiChannelGroupByHash.java","lineNumber":385,"sourceCode":"    {\n        if (currentPageBuilder != null) {\n            completedPagesMemorySize += currentPageBuilder.getRetainedSizeInBytes();\n            currentPageBuilder = currentPageBuilder.newPageBuilderLike();\n        }\n        else {\n            currentPageBuilder = new PageBuilder(types);\n        }\n\n        for (int i = 0; i < types.size(); i++) {\n            channelBuilders.get(i).add(currentPageBuilder.getBlockBuilder(i));\n        }\n    }\n\n    private boolean tryRehash()\n    {\n        long newCapacityLong = hashCapacity * 2L;\n        if (newCapacityLong > Integer.MAX_VALUE) {\n            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, \"Size of hash table cannot exceed 1 billion entries\");\n        }\n        int newCapacity = toIntExact(newCapacityLong);\n\n        // An estimate of how much extra memory is needed before we can go ahead and expand the hash table.\n        // This includes the new capacity for groupAddressByHash, rawHashByHashPosition, groupIdsByHash, and groupAddressByGroupId as well as the size of the current page\n        preallocatedMemoryInBytes = newCapacity * (long) (Long.BYTES + Integer.BYTES + Byte.BYTES) +\n                calculateMaxFill(newCapacity) * Long.BYTES +\n                currentPageSizeInBytes;\n        if (!updateMemory.update()) {\n            // reserved memory but has exceeded the limit\n            return false;\n        }\n\n        expectedHashCollisions += estimateNumberOfHashCollisions(getGroupCount(), hashCapacity);\n\n        int newMask = newCapacity - 1;\n        long[] newKey = new long[newCapacity];\n        byte[] rawHashes = new byte[newCapacity];","sourceCodeStart":367,"sourceCodeEnd":403,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-main-base/src/main/java/com/facebook/presto/operator/MultiChannelGroupByHash.java#L367-L403","documentation":"MultiChannelGroupByHash.tryRehash doubles the hash table capacity; if the new capacity would exceed Integer.MAX_VALUE the group-by hash cannot grow further, so it throws GENERIC_INSUFFICIENT_RESOURCES declaring the hard limit of ~1 billion entries. This is a resource/limits error, typically from extremely high group counts.","triggerScenarios":"tryRehash (called from addNewGroup) when hashCapacity * 2 > Integer.MAX_VALUE, i.e. the group-by hash has grown to ~1 billion (2^30) entries and another group is added.","commonSituations":"Aggregations with hundreds of millions to billions of distinct grouping keys, runaway DISTINCT values, or a join/aggregate producing a cartesian explosion.","solutions":["Reduce cardinality: add coarser GROUP BY keys or pre-aggregate the input","Filter out unneeded rows/columns before the aggregation","Enable spill / increase memory so lower-cardinality plans execute, or split the query across partitions","Raise query limits is not possible for this hard cap; redesign the query instead"],"exampleFix":"-- before\nSELECT customerId, sessionId, eventTime, count(*) FROM events GROUP BY customerId, sessionId, eventTime;\n-- after: bucket to reduce groups\nSELECT customerId, date_trunc('hour', eventTime), count(*) FROM events GROUP BY 1, 2;","handlingStrategy":"validation","validationCode":"-- estimate cardinality before running\nSELECT approx_distinct(groupKeyExpr) FROM source_table;\n-- if result approaches 1e9, restructure the query","typeGuard":null,"tryCatchPattern":"try { query(aggSql); } catch (PrestoException e) {\n    if (e.getErrorCode().getName().equals(\"GENERIC_INSUFFICIENT_RESOURCES\")) {\n        query(aggSqlWithCoarserGrouping); // fallback plan\n    } else { throw e; }\n}","preventionTips":["Use approx_distinct to sanity-check group cardinality before large aggregations","Pre-aggregate or bucket data upstream to reduce distinct keys","Avoid grouping on high-cardinality combinations (ids, timestamps at full precision)"],"tags":["memory","aggregation","group-by","resource-limit"],"backgroundTag":"hash-table-capacity-exceeded","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}