What causes "Failed to write json to file: %s" in apache/iceberg?

Inspect the chained cause (getCause) for the real storage error and fix that condition

apache/iceberg · error · RuntimeIOException

Failed to write json to file: %s

Error message

Failed to write json to file: %s

What it means

internalWrite serializes TableMetadata to JSON and streams it (optionally gzip-compressed) to an output file. Any IOException from the underlying FileIO stream is wrapped in RuntimeIOException with the output file location. This indicates the metadata JSON could not be written to storage — an I/O or filesystem-level failure, not a serialization bug.

Source

Thrown at core/src/main/java/org/apache/iceberg/TableMetadataParser.java:136

  public static void overwrite(TableMetadata metadata, OutputFile outputFile) {
    internalWrite(metadata, outputFile, true);
  }

  public static void write(TableMetadata metadata, OutputFile outputFile) {
    internalWrite(metadata, outputFile, false);
  }

  public static void internalWrite(
      TableMetadata metadata, OutputFile outputFile, boolean overwrite) {
    boolean isGzip = Codec.fromFileName(outputFile.location()) == Codec.GZIP;
    OutputStream stream = overwrite ? outputFile.createOrOverwrite() : outputFile.create();
    try (OutputStream ou = isGzip ? new GZIPOutputStream(stream) : stream;
        OutputStreamWriter writer = new OutputStreamWriter(ou, StandardCharsets.UTF_8);
        JsonGenerator generator = JsonUtil.factory().createGenerator(writer)) {
      toJson(metadata, generator);
      generator.flush();
    } catch (IOException e) {
      throw new RuntimeIOException(e, "Failed to write json to file: %s", outputFile.location());
    }
  }

  public static String getFileExtension(String codecName) {
    return getFileExtension(Codec.fromName(codecName));
  }

  public static String getFileExtension(Codec codec) {
    return codec.extension + ".metadata.json";
  }

  public static String getOldFileExtension(Codec codec) {
    // we have to be backward-compatible with .metadata.json.gz files
    return ".metadata.json" + codec.extension;
  }

  public static String toJson(TableMetadata metadata) {
    try (StringWriter writer = new StringWriter();

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the chained cause (getCause) for the real storage error and fix that condition
Retry the commit — Iceberg commits are safe to retry; a transient object-store blip is the most common cause
Verify write credentials/permissions on the table's metadata location
Check storage quota/disk space and network connectivity to the object store

Example fix

// before
TableMetadataParser.overwrite(metadata, outputFile); // throws on transient S3 503
// after
Tasks.foreach(metadata)
    .retry(3)
    .exponentialBackoff(100, 1000, 10000)
    .throwFailureWhenFinished()
    .run(m -> TableMetadataParser.overwrite(m, outputFile));

Defensive patterns

Strategy: retry

Validate before calling

// pre-check storage writability
try (OutputStream out = fileIO.newOutputFile(tmpLoc).createOrOverwrite()) { out.write(1); }

Try / catch

try { TableMetadataParser.overwrite(metadata, outputFile); }
catch (RuntimeIOException e) {
  LOG.error("Metadata write failed for {}: cause={}", outputFile.location(), e.getCause(), e);
  throw e;
}

Prevention

Retry commits with Tasks.foreach retry + exponential backoff for transient object-store errors
Monitor storage quota, disk space, and bucket permissions
Verify FileIO credentials before long-running write jobs
Inspect e.getCause() to distinguish permission vs connectivity vs quota failures

When it happens

Trigger: Any table commit (write/overwrite) that creates a new metadata JSON via TableMetadataParser.overwrite or write when the underlying OutputStream raises IOException: storage outage, quota exceeded, permission denied on the metadata directory, network failure to object store.

Common situations: S3/GCS/HDFS transient outages or throttling during commits; wrong credentials or missing write permission on the table location; disk full on local/HDFS paths; anti-virus or permission churn on local filesystems.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/e276037f07516e01. Report an issue: GitHub.