apache/iceberg · critical · UncheckedIOException

Failed writing offset to: %s

Error message

Failed writing offset to: %s

What it means

SparkMicroBatchStream writes the current StreamingOffset as JSON to a file under the initial offset location so Spark Structured Streaming can track scan progress. This UncheckedIOException wraps any IOException that occurs while creating or writing that offset file. It indicates the streaming query cannot persist its checkpoint offset, so the batch cannot proceed safely.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:301

      }

      table.refresh();
      StreamingOffset offset = MicroBatchUtils.determineStartingOffset(table, fromTimestamp);

      OutputFile outputFile = io.newOutputFile(initialOffsetLocation);
      writeOffset(offset, outputFile);

      return offset;
    }

    private void writeOffset(StreamingOffset offset, OutputFile file) {
      try (OutputStream outputStream = file.create()) {
        BufferedWriter writer =
            new BufferedWriter(new OutputStreamWriter(outputStream, StandardCharsets.UTF_8));
        writer.write(offset.json());
        writer.flush();
      } catch (IOException ioException) {
        throw new UncheckedIOException(
            String.format("Failed writing offset to: %s", initialOffsetLocation), ioException);
      }
    }

    private StreamingOffset readOffset(InputFile file) {
      try (InputStream in = file.newStream()) {
        return StreamingOffset.fromJson(in);
      } catch (IOException ioException) {
        throw new UncheckedIOException(
            String.format("Failed reading offset from: %s", initialOffsetLocation), ioException);
      }
    }
  }
}

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Check that the offset/checkpoint location exists and is writable by the job's identity (fs permissions, IAM/S3 bucket policy).
  2. Inspect the wrapped IOException (cause) for the root filesystem error and fix it (disk full, connectivity, throttling).
  3. Retry the streaming query after resolving transient storage outages; clear a corrupt/partial offset file only after verifying checkpoint semantics.
  4. Pin/stabilize the underlying filesystem (Hadoop aws/hdfs versions, retry settings) so file.create() is resilient.

Example fix

// before: raw UncheckedIOException surfaces from stream init
StreamingOffset offset = stream.initialOffset();
// after: pre-validate offset location writability
Path offsetPath = new Path(initialOffsetLocation);
FileSystem fs = offsetPath.getFileSystem(conf);
Preconditions.checkState(fs.mkdirs(offsetPath.getParent()), "Offset dir not creatable: %s", offsetPath.getParent());
StreamingOffset offset = stream.initialOffset();
Defensive patterns

Strategy: validation

Validate before calling

Path offsetPath = new Path(initialOffsetLocation);
FileSystem fs = offsetPath.getFileSystem(conf);
if (!fs.exists(offsetPath.getParent())) {
  fs.mkdirs(offsetPath.getParent());
}
FileStatus st = fs.getFileStatus(offsetPath.getParent());
// verify write access before starting the stream

Try / catch

try { offset = stream.initialOffset(); } catch (UncheckedIOException e) {
  LOG.error("Offset write failed at {}: {}", initialOffsetLocation, e.getCause());
  throw new IllegalStateException("Fix offset location access/retry storage", e);
}

Prevention

When it happens

Trigger: initialOffset() calls writeOffset and the underlying FileSystem (file.create()) fails to create or write the offset file — e.g. permission denied on the checkpoint/offset directory, HDFS/S3 transient outage, disk full, or the offset location was deleted between checks.

Common situations: S3/HDFS transient unavailability during a streaming query start; checkpoint directory permissions changed or the offset file made read-only; container disk pressure; offset location pointing to a wrong or removed path after a config change.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/4c31d90b2b615cb9. Report an issue: GitHub.