{"record":{"id":"6febe58f08b3ec1b","repo":"apache/iceberg","slug":"failed-to-write-data-manifests","errorCode":null,"errorMessage":"Failed to write data manifests","messagePattern":"Failed to write data manifests","errorType":"exception","errorClass":"RuntimeIOException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/SnapshotProducer.java","lineNumber":770,"sourceCode":"    }\n\n    if (clearManifests && anyDeleted) {\n      manifests.clear();\n    }\n  }\n\n  private List<ManifestFile> writeDataFileGroup(\n      Collection<DataFile> files, Long dataSeq, PartitionSpec spec) {\n    RollingManifestWriter<DataFile> writer = newRollingManifestWriter(spec);\n\n    try (RollingManifestWriter<DataFile> closableWriter = writer) {\n      if (dataSeq != null) {\n        files.forEach(file -> closableWriter.add(file, dataSeq));\n      } else {\n        files.forEach(closableWriter::add);\n      }\n    } catch (IOException e) {\n      throw new RuntimeIOException(e, \"Failed to write data manifests\");\n    }\n\n    return writer.toManifestFiles();\n  }\n\n  protected List<ManifestFile> writeDeleteManifests(\n      Collection<DeleteFile> files, PartitionSpec spec) {\n    int groupCount = manifestWriterCount(writePoolParallelism, files.size());\n    return ManifestFiles.writeParallel(\n        files, groupCount, writePool(), group -> writeDeleteFileGroup(group, spec));\n  }\n\n  private List<ManifestFile> writeDeleteFileGroup(\n      Collection<DeleteFile> files, PartitionSpec spec) {\n    RollingManifestWriter<DeleteFile> writer = newRollingDeleteManifestWriter(spec);\n\n    try (RollingManifestWriter<DeleteFile> closableWriter = writer) {\n      for (DeleteFile file : files) {","sourceCodeStart":752,"sourceCodeEnd":788,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L752-L788","documentation":"While producing a snapshot, SnapshotProducer writes one manifest file per group of data files via a ManifestWriter. Any IOException while writing these Avro manifests is wrapped as RuntimeIOException with this message. The snapshot cannot be produced because the data-file listing metadata was not persisted.","triggerScenarios":"newFastAppend().appendFile(...).commit(), newOverwrite, or rewriteDataFiles commits when the FileIO fails to create/write the data manifest Avro files in the table metadata location.","commonSituations":"Expired cloud credentials mid-job; wrong warehouse/metadata path; disk full on the metadata volume; object store rate limits when writing many manifests in parallel (workerPool); HDFS lease/replication issues.","solutions":["Verify the table's metadata location is writable and credentials for FileIO are valid.","Retry the operation; transient storage/network faults are the most common cause.","Check e.getCause() (the original IOException) for the storage-specific error code.","Reduce manifest parallelism or manifest target size if hitting throttling or many-small-file pressure."],"exampleFix":"// before\ntable.newFastAppend()\n    .appendFile(dataFile)\n    .commit(); // RuntimeIOException: Failed to write data manifests (quota exceeded)\n// after\nTableProperties.writeProperties(table)\n    .set(TableProperties.MANIFEST_TARGET_SIZE_BYTES, \"134217728\");\ntable.refresh();\ntable.newFastAppend().appendFile(dataFile).commit(); // after freeing quota / fixing creds","handlingStrategy":"try-catch","validationCode":"// pre-flight: ensure the metadata dir is writable\nString meta = ((BaseTable) table).operations().current().metadataFileLocation();\nString dir = meta.substring(0, meta.lastIndexOf('/'));\nif (!table.io() instanceof SupportsPrefixOperations) { /* attempt probe create/delete */ }","typeGuard":null,"tryCatchPattern":"try {\n  producer.commit();\n} catch (RuntimeIOException e) {\n  LOGGER.error(\"manifest write failed: {}\", e.getCause(), e);\n  throw new CommitStateUnknownException(e);\n}","preventionTips":["Keep credentials for FileIO refreshed (instance profiles, workload identity).","Cap manifest writer parallelism to avoid throttling during big commits.","Ensure the metadata location is not the same volume as data writes competing for space.","Test commits against the target store in CI with representative file counts."],"tags":["io","manifest","commit","runtime-io-exception"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}