{"record":{"id":"4e40b6e07037bdce","repo":"apache/iceberg","slug":"unable-to-close-the-manifest-writer-s-4e40b6","errorCode":null,"errorMessage":"Unable to close the manifest writer: %s","messagePattern":"Unable to close the manifest writer: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java","lineNumber":235,"sourceCode":"      FileIO io = new HadoopFileIO(conf.get());\n      TaskContext ctx = TaskContext.get();\n      String suffix =\n          String.format(\n              Locale.ROOT,\n              \"stage-%d-task-%d-manifest-%s\",\n              ctx.stageId(),\n              ctx.taskAttemptId(),\n              UUID.randomUUID());\n      Path location = new Path(basePath, suffix);\n      String outputPath = FileFormat.AVRO.addExtension(location.toString());\n      OutputFile outputFile = io.newOutputFile(outputPath);\n      ManifestWriter<DataFile> writer =\n          ManifestFiles.write(formatVersion, spec, outputFile, snapshotId);\n\n      try (ManifestWriter<DataFile> writerRef = writer) {\n        fileTuples.forEachRemaining(fileTuple -> writerRef.add(fileTuple._2));\n      } catch (IOException e) {\n        throw SparkExceptionUtil.toUncheckedException(\n            e, \"Unable to close the manifest writer: %s\", outputPath);\n      }\n\n      ManifestFile manifestFile = writer.toManifestFile();\n      return ImmutableList.of(manifestFile).iterator();\n    } else {\n      return Collections.emptyIterator();\n    }\n  }\n\n  /**\n   * Import files from an existing Spark table to an Iceberg table.\n   *\n   * <p>The import uses the Spark session to get table metadata. It assumes no operation is going on\n   * the original and target table and thus is not thread-safe.\n   *\n   * @param spark a Spark session\n   * @param sourceTableIdent an identifier of the source Spark table","sourceCodeStart":217,"sourceCodeEnd":253,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java#L217-L253","documentation":"Thrown by SparkTableUtil.buildManifest when writing imported data files into an Iceberg manifest fails with an IOException while the manifest writer is being used/closed. The manifest file being written to the given output path could not be produced, so the import aborts. This is an I/O problem on the target filesystem, not a catalog or SQL problem.","triggerScenarios":"Calling importUnpartitionedSparkTable/importPartitionedSparkTable during migration when the manifest output file (under the Iceberg table's write location) cannot be written — unwritable storage, full disk, broken HDFS/S3 credentials, or an IOException while streaming DataFiles into the writer.","commonSituations":"Target table location on HDFS with insufficient permissions; S3/GCS credentials expiring mid-import for large imports; disk-full on local warehouse in tests; network blips to object storage when creating/writing the manifest file.","solutions":["Check permissions and quota on the target table's write location; verify the process can create files there.","Verify storage credentials/config (Hadoop creds, S3 access keys, token expiry) and re-run after refreshing them.","Check disk space and filesystem health; retry the import after freeing space or fixing the storage outage.","If transient, re-run the import — the append commits only after manifests are built, so a failed attempt leaves no partial Iceberg snapshot."],"exampleFix":"// before (fails: S3 creds expired mid-import)\nSparkTableUtil.importUnpartitionedSparkTable(spark, ident, table);\n// after: refresh credentials / ensure fs perms before import\nspark.conf().set(\"spark.hadoop.fs.s3a.access.key\", newKey);\nspark.conf().set(\"spark.hadoop.fs.s3a.secret.key\", newSecret);\nSparkTableUtil.importUnpartitionedSparkTable(spark, ident, table);","handlingStrategy":"try-catch","validationCode":"// ensure the table location is writable before import\nassertWritable(targetTable.location()); // e.g. create/delete a temp object via FileIO\ntable.io().deleteFile(tempProbePath);","typeGuard":null,"tryCatchPattern":"try {\n  SparkTableUtil.importUnpartitionedSparkTable(spark, ident, target);\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Unable to close the manifest writer\")) {\n    // inspect cause for IOException; verify storage creds/permissions/space, then retry\n  } else throw e;\n}","preventionTips":["Pre-flight write test against the table's storage location","Refresh/extend object-storage credentials for long imports","Monitor disk space and quota on the warehouse directory","Rely on atomic commit: failed imports leave no snapshot, safe to retry"],"tags":["spark","io","manifest"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}