{"record":{"id":"d1994d0fa4d811a8","repo":"prestodb/presto","slug":"hive-too-many-open-partitions","errorCode":"HIVE_TOO_MANY_OPEN_PARTITIONS","errorMessage":"Exceeded limit of %s open writers for partitions/buckets","messagePattern":"Exceeded limit of (.+?) open writers for partitions/buckets","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-hive/src/main/java/com/facebook/presto/hive/HivePageSink.java","lineNumber":396,"sourceCode":"            HiveWriter writer = writers.get(index);\n\n            long currentWritten = writer.getWrittenBytes();\n            long currentMemory = writer.getSystemMemoryUsage();\n\n            writer.append(pageForWriter);\n\n            writtenBytes += (writer.getWrittenBytes() - currentWritten);\n            systemMemoryUsage += (writer.getSystemMemoryUsage() - currentMemory);\n        }\n    }\n\n    private int[] getWriterIndexes(Page page)\n    {\n        Page partitionColumns = extractColumns(page, partitionColumnsInputIndex);\n        Block bucketBlock = buildBucketBlock(page);\n        int[] writerIndexes = pagePartitioner.partitionPage(partitionColumns, bucketBlock);\n        if (pagePartitioner.getMaxIndex() >= maxOpenWriters) {\n            throw new PrestoException(HIVE_TOO_MANY_OPEN_PARTITIONS, format(\"Exceeded limit of %s open writers for partitions/buckets\", maxOpenWriters));\n        }\n\n        // expand writers list to new size\n        while (writers.size() <= pagePartitioner.getMaxIndex()) {\n            writers.add(null);\n        }\n\n        // create missing writers\n        for (int position = 0; position < page.getPositionCount(); position++) {\n            int writerIndex = writerIndexes[position];\n            if (writers.get(writerIndex) != null) {\n                continue;\n            }\n\n            OptionalInt bucketNumber = OptionalInt.empty();\n            if (bucketBlock != null) {\n                bucketNumber = OptionalInt.of(bucketBlock.getInt(position));\n            }","sourceCodeStart":378,"sourceCodeEnd":414,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-hive/src/main/java/com/facebook/presto/hive/HivePageSink.java#L378-L414","documentation":"HivePageSink keeps one open writer per (partition, bucket) combination. When a page partitions to more distinct writers than hive.max-open-writers (maxOpenWriters), the sink refuses to proceed and throws HIVE_TOO_MANY_OPEN_PARTITIONS to bound memory/file-handle usage.","triggerScenarios":"appendPage -> getWriterIndexes computes pagePartitioner.partitionPage and getMaxIndex() >= maxOpenWriters: the page targets more distinct partitions or buckets than the configured limit.","commonSituations":"Writing to highly cardinality partition columns (e.g. per-user or per-timestamp partitions), INSERT into a table with many partitions with default max-open-writers=100, skewed grouping spreading rows over many buckets.","solutions":["Raise the session/catalog property hive.max-open-writers to exceed the number of distinct partitions/buckets written in one query.","Reduce partition cardinality of the write (coarser partition columns, e.g. daily instead of per-minute).","Split the write into multiple INSERT statements each touching fewer partitions.","Pre-partition data upstream or use a two-phase write with intermediate table."],"exampleFix":"-- before\nINSERT INTO logs PARTITION (dt) SELECT ...; -- thousands of dt values\n-- after\nSET SESSION hive.max_open_writers = 5000;\n-- or write in ranges:\nINSERT INTO logs SELECT ... WHERE dt BETWEEN '2026-09-01' AND '2026-09-05';","handlingStrategy":"validation","validationCode":"-- estimate distinct partitions before writing\nSELECT count(DISTINCT dt) FROM staging_source;\n-- compare with:\nSHOW SESSION LIKE '%max_open_writers%';","typeGuard":null,"tryCatchPattern":"catch (PrestoException e) {\n    if (\"HIVE_TOO_MANY_OPEN_PARTITIONS\".equals(e.getErrorCode().getName())) {\n        // raise hive.max_open_writers or reduce write cardinality, then retry\n    }\n}","preventionTips":["Before large INSERTs, count distinct partition-key values and size max-open-writers above it.","Prefer coarser partition keys for high-cardinality data.","Never set max-open-writers extremely high without checking executor memory/file-handle limits.","Batch writes into multiple INSERTs partitioned by range when cardinality is huge."],"tags":["hive","sink","partitions","limit-exceeded","configuration"],"backgroundTag":"too-many-open-writers","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}