What causes "Failed writing offset to: %s" in apache/iceberg?

Verify the offset location directory exists and is writable by the Spark driver/executor, creating it if needed

apache/iceberg · error · UncheckedIOException

Failed writing offset to: %s

Error message

Failed writing offset to: %s

What it means

SparkMicroBatchStream persists the current StreamingOffset as a JSON file at the initial offset location so streaming queries can resume. If writing that offset file fails with an IOException, it is rethrown as UncheckedIOException with the target location. A failed offset write means the stream cannot checkpoint its progress.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:303

      }

      table.refresh();
      StreamingOffset offset = MicroBatchUtils.determineStartingOffset(table, fromTimestamp);

      OutputFile outputFile = io.newOutputFile(initialOffsetLocation);
      writeOffset(offset, outputFile);

      return offset;
    }

    private void writeOffset(StreamingOffset offset, OutputFile file) {
      try (OutputStream outputStream = file.create()) {
        BufferedWriter writer =
            new BufferedWriter(new OutputStreamWriter(outputStream, StandardCharsets.UTF_8));
        writer.write(offset.json());
        writer.flush();
      } catch (IOException ioException) {
        throw new UncheckedIOException(
            String.format("Failed writing offset to: %s", initialOffsetLocation), ioException);
      }
    }

    private StreamingOffset readOffset(InputFile file) {
      try (InputStream in = file.newStream()) {
        return StreamingOffset.fromJson(in);
      } catch (IOException ioException) {
        throw new UncheckedIOException(
            String.format("Failed reading offset from: %s", initialOffsetLocation), ioException);
      }
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

Verify the offset location directory exists and is writable by the Spark driver/executor, creating it if needed
Check storage backend health and credentials (HDFS availability, S3 keys/roles, disk space)
Inspect the chained IOException cause for the root storage error and fix that; then restart the streaming query from a valid checkpoint

Defensive patterns

Strategy: try-catch

Validate before calling

File offsetDir = new File(initialOffsetLocation);
if (!offsetDir.getParentFile().canWrite()) throw new IllegalStateException("Offset dir not writable");

Try / catch

try {
  stream.writeOffset(offset);
} catch (UncheckedIOException e) {
  LOG.error("Offset write failed at {}", e.getMessage(), e.getCause());
  // remediate storage, then restart from checkpoint
}

Prevention

Provision and monitor the offset/checkpoint directory with correct permissions
Monitor disk space and object-store health for the streaming location
Keep storage credentials valid for the lifetime of the streaming query

When it happens

Trigger: Writing the streaming offset file fails due to an IOException — unwritable/missing directory, out-of-space filesystem, permissions, or FileIO returning a broken output stream at the initialOffsetLocation.

Common situations: Checkpoint/offset directory deleted or permissions changed between runs; HDFS/S3 outages or token expiry when creating the offset file; disk full on the driver or local FS; misconfigured warehouse path.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/acc48f7670304c5f. Report an issue: GitHub.