{"record":{"id":"08a572555bec01e1","repo":"prestodb/presto","slug":"iceberg-too-many-open-partitions","errorCode":"ICEBERG_TOO_MANY_OPEN_PARTITIONS","errorMessage":"Exceeded limit of %s open writers for partitions","messagePattern":"Exceeded limit of (.+?) open writers for partitions","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergPageSink.java","lineNumber":412,"sourceCode":"    }\n\n    private boolean isOmittedInsertColumn(IcebergColumnHandle column)\n    {\n        return !insertedColumns.isEmpty() && !insertedColumns.contains(column.getName());\n    }\n\n    private Block fillBlockWithDefault(Block block, IcebergColumnHandle column)\n    {\n        Object writeDefaultValue = deserializeIcebergValue(column.getType(), column.getWriteDefaultValue().get(), column.getName());\n        return RunLengthEncodedBlock.create(column.getType(), writeDefaultValue, block.getPositionCount());\n    }\n\n    private int[] getWriterIndexes(Page page)\n    {\n        int[] writerIndexes = pagePartitioner.partitionPage(page);\n\n        if (pagePartitioner.getMaxIndex() >= maxOpenWriters) {\n            throw new PrestoException(ICEBERG_TOO_MANY_OPEN_PARTITIONS, format(\"Exceeded limit of %s open writers for partitions\", maxOpenWriters));\n        }\n\n        // expand writers list to new size\n        while (writers.size() <= pagePartitioner.getMaxIndex()) {\n            writers.add(null);\n        }\n\n        // create missing writers\n        Page transformedPage = pagePartitioner.getTransformedPage();\n        for (int position = 0; position < page.getPositionCount(); position++) {\n            int writerIndex = writerIndexes[position];\n            WriteContext writer = writers.get(writerIndex);\n            if (writer != null) {\n                continue;\n            }\n\n            Optional<PartitionData> partitionData = getPartitionData(pagePartitioner.getColumns(), transformedPage, position);\n","sourceCodeStart":394,"sourceCodeEnd":430,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergPageSink.java#L394-L430","documentation":"getWriterIndexes partitions each incoming page and opens one Iceberg writer per distinct partition value. If a page contains more distinct partition values than maxOpenWriters (iceberg.max-open-writers / writer scaling limits), the sink would need more concurrent files than allowed, so it throws ICEBERG_TOO_MANY_OPEN_PARTITIONS to protect memory and file-handle limits.","triggerScenarios":"INSERT/CTAS into a partitioned Iceberg table where a single page/stage of data touches >= maxOpenWriters distinct partitions — e.g. high-cardinality partition columns (date+hour+region), unpartitioned-like data spread over thousands of date partitions, or a configured max-open-writers lower than the data's partition cardinality.","commonSituations":"Partitioning by a high-cardinality column (user_id, timestamp at second granularity); writing a backfill spanning many days with max-open-writers left at default; admin lowered the limit to reduce file counts but queries still fan out widely; page sizes reworked so more partitions co-occur per page.","solutions":["Increase iceberg.max-open-writers session/catalog property to exceed the number of distinct partitions written concurrently.","Reduce partition cardinality: re-partition the table on coarser keys (e.g. day instead of hour) via a new table + INSERT, or add bucket/truncate transforms.","Rewrite the load to write in batches that cover fewer partitions per query (e.g. loop day by day) so each statement stays under the writer limit.","If using sorted/spill-capable writer options in your connector version, enable writer buffering so fewer writers are open simultaneously."],"exampleFix":"-- before\nINSERT INTO events SELECT * FROM raw_events; -- spans 2000 day-partitions\n-- after\nSET SESSION iceberg.max_open_writers = 2500;\nINSERT INTO events SELECT * FROM raw_events;\n-- or better: partition coarser\nCREATE TABLE events_day WITH (partitioning = ARRAY['day(ts)']) AS SELECT * FROM raw_events;","handlingStrategy":"try-catch","validationCode":"-- estimate distinct partitions to be written before inserting\nSELECT COUNT(DISTINCT day(ts)) FROM raw_events;  -- must be < iceberg.max_open_writers","typeGuard":null,"tryCatchPattern":"try { execute(insertSql); }\ncatch (PrestoException e) {\n  if (e.getErrorCode().getName().equals(\"ICEBERG_TOO_MANY_OPEN_PARTITIONS\")) {\n    execute(\"SET SESSION iceberg.max_open_writers = 1000\");\n    execute(insertSql); // or split the write by partition range\n  } else throw e;\n}","preventionTips":["Set iceberg.max-open-writers above the max distinct partitions any single query will touch.","Prefer coarse/transformed partition keys (day(), bucket(), truncate()) over high-cardinality columns.","Chunk large backfills so each statement writes a bounded set of partitions.","Monitor writer memory since raising max-open-writers increases heap usage."],"tags":["iceberg","partitions","writers","resource-limit"],"backgroundTag":"too-many-open-partitions","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}