apache/iceberg · error · RuntimeIOException

Failed to write data manifests

Error message

Failed to write data manifests

What it means

While producing a snapshot, SnapshotProducer writes one manifest file per group of data files via a ManifestWriter. Any IOException while writing these Avro manifests is wrapped as RuntimeIOException with this message. The snapshot cannot be produced because the data-file listing metadata was not persisted.

Source

Thrown at core/src/main/java/org/apache/iceberg/SnapshotProducer.java:770

    }

    if (clearManifests && anyDeleted) {
      manifests.clear();
    }
  }

  private List<ManifestFile> writeDataFileGroup(
      Collection<DataFile> files, Long dataSeq, PartitionSpec spec) {
    RollingManifestWriter<DataFile> writer = newRollingManifestWriter(spec);

    try (RollingManifestWriter<DataFile> closableWriter = writer) {
      if (dataSeq != null) {
        files.forEach(file -> closableWriter.add(file, dataSeq));
      } else {
        files.forEach(closableWriter::add);
      }
    } catch (IOException e) {
      throw new RuntimeIOException(e, "Failed to write data manifests");
    }

    return writer.toManifestFiles();
  }

  protected List<ManifestFile> writeDeleteManifests(
      Collection<DeleteFile> files, PartitionSpec spec) {
    int groupCount = manifestWriterCount(writePoolParallelism, files.size());
    return ManifestFiles.writeParallel(
        files, groupCount, writePool(), group -> writeDeleteFileGroup(group, spec));
  }

  private List<ManifestFile> writeDeleteFileGroup(
      Collection<DeleteFile> files, PartitionSpec spec) {
    RollingManifestWriter<DeleteFile> writer = newRollingDeleteManifestWriter(spec);

    try (RollingManifestWriter<DeleteFile> closableWriter = writer) {
      for (DeleteFile file : files) {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Verify the table's metadata location is writable and credentials for FileIO are valid.
2. Retry the operation; transient storage/network faults are the most common cause.
3. Check e.getCause() (the original IOException) for the storage-specific error code.
4. Reduce manifest parallelism or manifest target size if hitting throttling or many-small-file pressure.

Example fix

// before
table.newFastAppend()
    .appendFile(dataFile)
    .commit(); // RuntimeIOException: Failed to write data manifests (quota exceeded)
// after
TableProperties.writeProperties(table)
    .set(TableProperties.MANIFEST_TARGET_SIZE_BYTES, "134217728");
table.refresh();
table.newFastAppend().appendFile(dataFile).commit(); // after freeing quota / fixing creds

// pre-flight: ensure the metadata dir is writable
String meta = ((BaseTable) table).operations().current().metadataFileLocation();
String dir = meta.substring(0, meta.lastIndexOf('/'));
if (!table.io() instanceof SupportsPrefixOperations) { /* attempt probe create/delete */ }

try {
  producer.commit();
} catch (RuntimeIOException e) {
  LOGGER.error("manifest write failed: {}", e.getCause(), e);
  throw new CommitStateUnknownException(e);
}

When it happens

Trigger: newFastAppend().appendFile(...).commit(), newOverwrite, or rewriteDataFiles commits when the FileIO fails to create/write the data manifest Avro files in the table metadata location.

Common situations: Expired cloud credentials mid-job; wrong warehouse/metadata path; disk full on the metadata volume; object store rate limits when writing many manifests in parallel (workerPool); HDFS lease/replication issues.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failed to write delete manifests
• Failed to write manifest list file
• Failed to read manifest: %s
• Failed to read manifest file: %s
• Failed to validate replaced partitions

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/6febe58f08b3ec1b. Report an issue: GitHub.