{"record":{"id":"7ae6bbdb8e6d4be0","repo":"prestodb/presto","slug":"generic-insufficient-resources-7ae6bb","errorCode":"GENERIC_INSUFFICIENT_RESOURCES","errorMessage":"Size of hash table cannot exceed 1 billion entries","messagePattern":"Size of hash table cannot exceed 1 billion entries","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/histogram/SingleTypedHistogram.java","lineNumber":189,"sourceCode":"    }\n\n    private void addNewGroup(int hashPosition, int position, Block block, long count)\n    {\n        hashPositions.set(hashPosition, values.getPositionCount());\n        counts.set(values.getPositionCount(), count);\n        type.appendTo(block, position, values);\n\n        // increase capacity, if necessary\n        if (values.getPositionCount() >= maxFill) {\n            rehash();\n        }\n    }\n\n    private void rehash()\n    {\n        long newCapacityLong = hashCapacity * 2L;\n        if (newCapacityLong > Integer.MAX_VALUE) {\n            throw new PrestoException(GENERIC_INSUFFICIENT_RESOURCES, \"Size of hash table cannot exceed 1 billion entries\");\n        }\n        int newCapacity = (int) newCapacityLong;\n\n        int newMask = newCapacity - 1;\n        IntBigArray newHashPositions = new IntBigArray(-1);\n        newHashPositions.ensureCapacity(newCapacity);\n\n        for (int i = 0; i < values.getPositionCount(); i++) {\n            // find an empty slot for the address\n            int hashPosition = getBucketId(TypeUtils.hashPosition(type, values, i), newMask);\n\n            while (newHashPositions.get(hashPosition) != -1) {\n                hashPosition = (hashPosition + 1) & newMask;\n            }\n\n            // record the mapping\n            newHashPositions.set(hashPosition, i);\n        }","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-main-base/src/main/java/com/facebook/presto/operator/aggregation/histogram/SingleTypedHistogram.java#L171-L207","documentation":"The typed histogram aggregation maintains an open-addressing hash table mapping group IDs to positions. Its capacity is stored as an int, so when doubling during rehash would exceed Integer.MAX_VALUE the aggregation aborts with GENERIC_INSUFFICIENT_RESOURCES, since a histogram with more than ~1 billion distinct entries cannot fit in the in-memory structure. This is a hard resource limit of the implementation, not a query bug.","triggerScenarios":"Calling the histogram() aggregation (e.g. histogram(col)) on a column whose number of distinct values is so large that SingleTypedHistogram.rehash(), invoked from addNewGroup as new distinct groups are added, would need to double hashCapacity past Integer.MAX_VALUE (capacity * 2L > Integer.MAX_VALUE).","commonSituations":"Aggregating histogram over a very high-cardinality column (e.g. user IDs, device IDs, request IDs) in a large table; a join or filter bug causing near-unique keys; attempting to build a histogram per group where each group still has billions of distinct values.","solutions":["Reduce the number of distinct values fed into histogram() by pre-aggregating, bucketing, or casting keys to a coarser granularity (e.g. truncate timestamps, bucket numeric IDs).","Add WHERE filters or GROUP BY additional columns so each histogram contains fewer distinct values.","If a full histogram is truly needed, extract distinct data in chunks or use approx_distinct / sketches instead of an exact histogram."],"exampleFix":"// before\nSELECT histogram(user_id) FROM events;\n// after\nSELECT histogram(bucketize(user_id)) -- or filter/limit cardinality\nSELECT histogram(date_trunc('hour', event_time)) FROM events WHERE dt = CURRENT_DATE;","handlingStrategy":"validation","validationCode":"-- Before running: ensure distinct count per histogram group is well under ~1e9\nSELECT count(DISTINCT key_col) FROM my_table WHERE <group_filter>;","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check cardinality with approx_distinct before using histogram() on large tables.","Bucket or truncate high-cardinality keys before aggregating.","Prefer sketches (approx_distinct, QDigest) over exact histograms at billion-scale."],"tags":["aggregation","memory-limit","histogram","resource-exhaustion"],"backgroundTag":"hash-table-capacity-exceeded","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}