apache/iceberg · error

Unable to close the manifest writer

Error message

Unable to close the manifest writer: %s

What it means

Thrown by SparkTableUtil.buildManifest when writing imported data files into an Iceberg manifest fails with an IOException while the manifest writer is being used/closed. The manifest file being written to the given output path could not be produced, so the import aborts. This is an I/O problem on the target filesystem, not a catalog or SQL problem.

Solutions

1. Check permissions and quota on the target table's write location; verify the process can create files there.
2. Verify storage credentials/config (Hadoop creds, S3 access keys, token expiry) and re-run after refreshing them.
3. Check disk space and filesystem health; retry the import after freeing space or fixing the storage outage.
4. If transient, re-run the import — the append commits only after manifests are built, so a failed attempt leaves no partial Iceberg snapshot.

Example fix

// before (fails: S3 creds expired mid-import)
SparkTableUtil.importUnpartitionedSparkTable(spark, ident, table);
// after: refresh credentials / ensure fs perms before import
spark.conf().set("spark.hadoop.fs.s3a.access.key", newKey);
spark.conf().set("spark.hadoop.fs.s3a.secret.key", newSecret);
SparkTableUtil.importUnpartitionedSparkTable(spark, ident, table);

// ensure the table location is writable before import
assertWritable(targetTable.location()); // e.g. create/delete a temp object via FileIO
table.io().deleteFile(tempProbePath);

try {
  SparkTableUtil.importUnpartitionedSparkTable(spark, ident, target);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unable to close the manifest writer")) {
    // inspect cause for IOException; verify storage creds/permissions/space, then retry
  } else throw e;
}

When it happens

Trigger: Calling importUnpartitionedSparkTable/importPartitionedSparkTable during migration when the manifest output file (under the Iceberg table's write location) cannot be written — unwritable storage, full disk, broken HDFS/S3 credentials, or an IOException while streaming DataFiles into the writer.

Common situations: Target table location on HDFS with insufficient permissions; S3/GCS credentials expiring mid-import for large imports; disk-full on local warehouse in tests; network blips to object storage when creating/writing the manifest file.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Delete failed for
• Failed reading offset from
• Failed reading offset from
• Failed reading offset from
• Failed to close changelog scan

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/4e40b6e07037bdce. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:235

      FileIO io = new HadoopFileIO(conf.get());
      TaskContext ctx = TaskContext.get();
      String suffix =
          String.format(
              Locale.ROOT,
              "stage-%d-task-%d-manifest-%s",
              ctx.stageId(),
              ctx.taskAttemptId(),
              UUID.randomUUID());
      Path location = new Path(basePath, suffix);
      String outputPath = FileFormat.AVRO.addExtension(location.toString());
      OutputFile outputFile = io.newOutputFile(outputPath);
      ManifestWriter<DataFile> writer =
          ManifestFiles.write(formatVersion, spec, outputFile, snapshotId);

      try (ManifestWriter<DataFile> writerRef = writer) {
        fileTuples.forEachRemaining(fileTuple -> writerRef.add(fileTuple._2));
      } catch (IOException e) {
        throw SparkExceptionUtil.toUncheckedException(
            e, "Unable to close the manifest writer: %s", outputPath);
      }

      ManifestFile manifestFile = writer.toManifestFile();
      return ImmutableList.of(manifestFile).iterator();
    } else {
      return Collections.emptyIterator();
    }
  }

  /**
   * Import files from an existing Spark table to an Iceberg table.
   *
   * <p>The import uses the Spark session to get table metadata. It assumes no operation is going on
   * the original and target table and thus is not thread-safe.
   *
   * @param spark a Spark session
   * @param sourceTableIdent an identifier of the source Spark table

View on GitHub (pinned to 86d9c8fc54)