{"record":{"id":"ca2b61e2a46c4fb4","repo":"prestodb/presto","slug":"generic-insufficient-resources-ca2b61","errorCode":"GENERIC_INSUFFICIENT_RESOURCES","errorMessage":"Size of hash table cannot exceed 2147483647 entries (%s)","messagePattern":"Size of hash table cannot exceed 2147483647 entries \\((.+?)\\)","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/histogram/ValueStore.java","lineNumber":127,"sourceCode":"                throw new PrestoException(NOT_SUPPORTED, e.getMessage(), e);\n            }\n        }\n    }\n\n    private int getBucketId(long valueHash, int mask)\n    {\n        return (int) (valueHash & mask);\n    }\n\n    @VisibleForTesting\n    void rehash()\n    {\n        ++rehashCount;\n\n        long newBucketCountLong = bucketCount * 2L;\n\n        if (newBucketCountLong > Integer.MAX_VALUE) {\n            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, \"Size of hash table cannot exceed \" + Integer.MAX_VALUE + \" entries (\" + newBucketCountLong + \")\");\n        }\n\n        int newBucketCount = (int) newBucketCountLong;\n        int newMask = newBucketCount - 1;\n\n        IntBigArray newBuckets = new IntBigArray(-1);\n\n        newBuckets.ensureCapacity(newBucketCount);\n\n        for (int i = 0; i < values.getPositionCount(); i++) {\n            long valueHash = valueHashes.get(i);\n            int bucketId = getBucketId(valueHash, newMask);\n            int probeCount = 1;\n\n            while (newBuckets.get(bucketId) != EMPTY_BUCKET) {\n                int probe = nextProbe(probeCount);\n\n                bucketId = nextBucketId(bucketId, newMask, probe);","sourceCodeStart":109,"sourceCodeEnd":145,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/histogram/ValueStore.java#L109-L145","documentation":"ValueStore.rehash doubles the internal bucket array of the hash-value store. Bucket count is an int, so when the doubled count would exceed Integer.MAX_VALUE the aggregation throws GENERIC_INSUFFICIENT_RESOURCES, reporting the would-be new capacity. This caps how many distinct hashed values a single ValueStore can hold (~2.1 billion buckets).","triggerScenarios":"addAndGetPosition inserting new value hashes into a histogram-style aggregation (ValueStore-backed) until the store's bucketCount * 2L exceeds Integer.MAX_VALUE during rehash.","commonSituations":"histogram() or similar exact aggregation over billions of distinct values; pathological inputs causing the value store to grow to its hard limit; under-estimated cardinality in test data versus production scale.","solutions":["Limit the number of distinct values per aggregation instance via filtering, bucketing, or additional GROUP BY keys.","Switch to approximate aggregations (approx_distinct, sketches) for very high cardinality.","Split the computation into multiple smaller queries/jobs and merge results outside Presto."],"exampleFix":"// before\nSELECT histogram(session_id) FROM clicks;\n// after\nSELECT histogram(session_id) FROM clicks WHERE dt = '2026-09-03'; -- and/or bucket keys first","handlingStrategy":"validation","validationCode":"SELECT approx_distinct(value_col) FROM my_table; -- keep result well below ~2e9","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Estimate cardinality first with approx_distinct.","Partition queries so each ValueStore handles a bounded set of distinct values.","Avoid exact histogram aggregations over near-unique keys (IDs, UUIDs)."],"tags":["aggregation","memory-limit","hash-table","resource-exhaustion"],"backgroundTag":"hash-table-capacity-exceeded","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}