apache/iceberg · error · UncheckedIOException

Failed writing offset to: ${initialOffsetLocation}

Error message

Failed writing offset to: ${initialOffsetLocation}

What it means

SparkMicroBatchStream persists streaming offsets as JSON files at initialOffsetLocation. If writing the offset file throws IOException, it is wrapped in UncheckedIOException with the target location. Losing this write aborts stream initialization/batch tracking.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:301

      }

      table.refresh();
      StreamingOffset offset = MicroBatchUtils.determineStartingOffset(table, fromTimestamp);

      OutputFile outputFile = io.newOutputFile(initialOffsetLocation);
      writeOffset(offset, outputFile);

      return offset;
    }

    private void writeOffset(StreamingOffset offset, OutputFile file) {
      try (OutputStream outputStream = file.create()) {
        BufferedWriter writer =
            new BufferedWriter(new OutputStreamWriter(outputStream, StandardCharsets.UTF_8));
        writer.write(offset.json());
        writer.flush();
      } catch (IOException ioException) {
        throw new UncheckedIOException(
            String.format("Failed writing offset to: %s", initialOffsetLocation), ioException);
      }
    }

    private StreamingOffset readOffset(InputFile file) {
      try (InputStream in = file.newStream()) {
        return StreamingOffset.fromJson(in);
      } catch (IOException ioException) {
        throw new UncheckedIOException(
            String.format("Failed reading offset from: %s", initialOffsetLocation), ioException);
      }
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Check the wrapped IOException cause for filesystem/permission details on initialOffsetLocation.
2. Verify the checkpoint location is writable with the running job's credentials.
3. Confirm storage availability/quotas, then restart the streaming query.

// ensure checkpoint location is creatable/writable before starting the stream
FileIO io = table.io();
io.checkOutputFile(initialOffsetLocation); // throws if unwritable

try { stream.initialOffset(); } catch (UncheckedIOException e) { log.error("offset write failed at {}: {}", loc, e.getCause()); throw e; }

When it happens

Trigger: initialOffset() calling writeOffset when the underlying FileIO cannot create/write the offset file (permissions, missing parent dir, storage outage, quota).

Common situations: Offset checkpoint location on S3/HDFS with permission or network problems; the checkpoint path deleted or bucket unavailable while a Spark structured-streaming query starts.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failed reading offset from: ${initialOffsetLocation}
• Table refresh failed
• Queue filling failed
• Failed to read StreamingOffset from json
• Failed to write StreamingOffset to json

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/46f220b6741500fa. Report an issue: GitHub.