prestodb/presto · error · PrestoException

HIVE_TOO_MANY_OPEN_PARTITIONS

HIVE_TOO_MANY_OPEN_PARTITIONS

Error message

Exceeded limit of %s open writers for partitions/buckets

What it means

HivePageSink keeps one open writer per (partition, bucket) combination. When a page partitions to more distinct writers than hive.max-open-writers (maxOpenWriters), the sink refuses to proceed and throws HIVE_TOO_MANY_OPEN_PARTITIONS to bound memory/file-handle usage.

Source

Thrown at presto-hive/src/main/java/com/facebook/presto/hive/HivePageSink.java:396

            HiveWriter writer = writers.get(index);

            long currentWritten = writer.getWrittenBytes();
            long currentMemory = writer.getSystemMemoryUsage();

            writer.append(pageForWriter);

            writtenBytes += (writer.getWrittenBytes() - currentWritten);
            systemMemoryUsage += (writer.getSystemMemoryUsage() - currentMemory);
        }
    }

    private int[] getWriterIndexes(Page page)
    {
        Page partitionColumns = extractColumns(page, partitionColumnsInputIndex);
        Block bucketBlock = buildBucketBlock(page);
        int[] writerIndexes = pagePartitioner.partitionPage(partitionColumns, bucketBlock);
        if (pagePartitioner.getMaxIndex() >= maxOpenWriters) {
            throw new PrestoException(HIVE_TOO_MANY_OPEN_PARTITIONS, format("Exceeded limit of %s open writers for partitions/buckets", maxOpenWriters));
        }

        // expand writers list to new size
        while (writers.size() <= pagePartitioner.getMaxIndex()) {
            writers.add(null);
        }

        // create missing writers
        for (int position = 0; position < page.getPositionCount(); position++) {
            int writerIndex = writerIndexes[position];
            if (writers.get(writerIndex) != null) {
                continue;
            }

            OptionalInt bucketNumber = OptionalInt.empty();
            if (bucketBlock != null) {
                bucketNumber = OptionalInt.of(bucketBlock.getInt(position));
            }

View on GitHub (pinned to 55bb57d202)

Solutions

Raise the session/catalog property hive.max-open-writers to exceed the number of distinct partitions/buckets written in one query.
Reduce partition cardinality of the write (coarser partition columns, e.g. daily instead of per-minute).
Split the write into multiple INSERT statements each touching fewer partitions.
Pre-partition data upstream or use a two-phase write with intermediate table.

Example fix

-- before
INSERT INTO logs PARTITION (dt) SELECT ...; -- thousands of dt values
-- after
SET SESSION hive.max_open_writers = 5000;
-- or write in ranges:
INSERT INTO logs SELECT ... WHERE dt BETWEEN '2026-09-01' AND '2026-09-05';

Defensive patterns

Strategy: validation

Validate before calling

-- estimate distinct partitions before writing
SELECT count(DISTINCT dt) FROM staging_source;
-- compare with:
SHOW SESSION LIKE '%max_open_writers%';

Try / catch

catch (PrestoException e) {
    if ("HIVE_TOO_MANY_OPEN_PARTITIONS".equals(e.getErrorCode().getName())) {
        // raise hive.max_open_writers or reduce write cardinality, then retry
    }
}

Prevention

Before large INSERTs, count distinct partition-key values and size max-open-writers above it.
Prefer coarser partition keys for high-cardinality data.
Never set max-open-writers extremely high without checking executor memory/file-handle limits.
Batch writes into multiple INSERTs partitioned by range when cardinality is huge.

When it happens

Trigger: appendPage -> getWriterIndexes computes pagePartitioner.partitionPage and getMaxIndex() >= maxOpenWriters: the page targets more distinct partitions or buckets than the configured limit.

Common situations: Writing to highly cardinality partition columns (e.g. per-user or per-timestamp partitions), INSERT into a table with many partitions with default max-open-writers=100, skewed grouping spreading rows over many buckets.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/d1994d0fa4d811a8. Report an issue: GitHub.