prestodb/presto · error · PrestoException

ICEBERG_TOO_MANY_OPEN_PARTITIONS

ICEBERG_TOO_MANY_OPEN_PARTITIONS

Error message

Exceeded limit of %s open writers for partitions

What it means

getWriterIndexes partitions each incoming page and opens one Iceberg writer per distinct partition value. If a page contains more distinct partition values than maxOpenWriters (iceberg.max-open-writers / writer scaling limits), the sink would need more concurrent files than allowed, so it throws ICEBERG_TOO_MANY_OPEN_PARTITIONS to protect memory and file-handle limits.

Source

Thrown at presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergPageSink.java:412

    }

    private boolean isOmittedInsertColumn(IcebergColumnHandle column)
    {
        return !insertedColumns.isEmpty() && !insertedColumns.contains(column.getName());
    }

    private Block fillBlockWithDefault(Block block, IcebergColumnHandle column)
    {
        Object writeDefaultValue = deserializeIcebergValue(column.getType(), column.getWriteDefaultValue().get(), column.getName());
        return RunLengthEncodedBlock.create(column.getType(), writeDefaultValue, block.getPositionCount());
    }

    private int[] getWriterIndexes(Page page)
    {
        int[] writerIndexes = pagePartitioner.partitionPage(page);

        if (pagePartitioner.getMaxIndex() >= maxOpenWriters) {
            throw new PrestoException(ICEBERG_TOO_MANY_OPEN_PARTITIONS, format("Exceeded limit of %s open writers for partitions", maxOpenWriters));
        }

        // expand writers list to new size
        while (writers.size() <= pagePartitioner.getMaxIndex()) {
            writers.add(null);
        }

        // create missing writers
        Page transformedPage = pagePartitioner.getTransformedPage();
        for (int position = 0; position < page.getPositionCount(); position++) {
            int writerIndex = writerIndexes[position];
            WriteContext writer = writers.get(writerIndex);
            if (writer != null) {
                continue;
            }

            Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), transformedPage, position);

View on GitHub (pinned to 55bb57d202)

Solutions

1. Increase iceberg.max-open-writers session/catalog property to exceed the number of distinct partitions written concurrently.
2. Reduce partition cardinality: re-partition the table on coarser keys (e.g. day instead of hour) via a new table + INSERT, or add bucket/truncate transforms.
3. Rewrite the load to write in batches that cover fewer partitions per query (e.g. loop day by day) so each statement stays under the writer limit.
4. If using sorted/spill-capable writer options in your connector version, enable writer buffering so fewer writers are open simultaneously.

Example fix

-- before
INSERT INTO events SELECT * FROM raw_events; -- spans 2000 day-partitions
-- after
SET SESSION iceberg.max_open_writers = 2500;
INSERT INTO events SELECT * FROM raw_events;
-- or better: partition coarser
CREATE TABLE events_day WITH (partitioning = ARRAY['day(ts)']) AS SELECT * FROM raw_events;

-- estimate distinct partitions to be written before inserting
SELECT COUNT(DISTINCT day(ts)) FROM raw_events;  -- must be < iceberg.max_open_writers

try { execute(insertSql); }
catch (PrestoException e) {
  if (e.getErrorCode().getName().equals("ICEBERG_TOO_MANY_OPEN_PARTITIONS")) {
    execute("SET SESSION iceberg.max_open_writers = 1000");
    execute(insertSql); // or split the write by partition range
  } else throw e;
}

When it happens

Trigger: INSERT/CTAS into a partitioned Iceberg table where a single page/stage of data touches >= maxOpenWriters distinct partitions — e.g. high-cardinality partition columns (date+hour+region), unpartitioned-like data spread over thousands of date partitions, or a configured max-open-writers lower than the data's partition cardinality.

Common situations: Partitioning by a high-cardinality column (user_id, timestamp at second granularity); writing a backfill spanning many days with max-open-writers left at default; admin lowered the limit to reduce file counts but queries still fan out widely; page sizes reworked so more partitions co-occur per page.

Related errors

• Not a Hive table:
• HIVE_TOO_MANY_OPEN_PARTITIONS
• HIVE_EXCEEDED_PARTITION_LIMIT
• HIVE_EXCEEDED_SPLIT_BUFFERING_LIMIT
• HIVE_UNKNOWN_ERROR

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/08a572555bec01e1. Report an issue: GitHub.