{"record":{"id":"46f220b6741500fa","repo":"apache/iceberg","slug":"failed-writing-offset-to-initialoffsetlocation","errorCode":null,"errorMessage":"Failed writing offset to: ${initialOffsetLocation}","messagePattern":"Failed writing offset to: (.+?)","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java","lineNumber":301,"sourceCode":"      }\n\n      table.refresh();\n      StreamingOffset offset = MicroBatchUtils.determineStartingOffset(table, fromTimestamp);\n\n      OutputFile outputFile = io.newOutputFile(initialOffsetLocation);\n      writeOffset(offset, outputFile);\n\n      return offset;\n    }\n\n    private void writeOffset(StreamingOffset offset, OutputFile file) {\n      try (OutputStream outputStream = file.create()) {\n        BufferedWriter writer =\n            new BufferedWriter(new OutputStreamWriter(outputStream, StandardCharsets.UTF_8));\n        writer.write(offset.json());\n        writer.flush();\n      } catch (IOException ioException) {\n        throw new UncheckedIOException(\n            String.format(\"Failed writing offset to: %s\", initialOffsetLocation), ioException);\n      }\n    }\n\n    private StreamingOffset readOffset(InputFile file) {\n      try (InputStream in = file.newStream()) {\n        return StreamingOffset.fromJson(in);\n      } catch (IOException ioException) {\n        throw new UncheckedIOException(\n            String.format(\"Failed reading offset from: %s\", initialOffsetLocation), ioException);\n      }\n    }\n  }\n}\n","sourceCodeStart":283,"sourceCodeEnd":316,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java#L283-L316","documentation":"SparkMicroBatchStream persists streaming offsets as JSON files at initialOffsetLocation. If writing the offset file throws IOException, it is wrapped in UncheckedIOException with the target location. Losing this write aborts stream initialization/batch tracking.","triggerScenarios":"initialOffset() calling writeOffset when the underlying FileIO cannot create/write the offset file (permissions, missing parent dir, storage outage, quota).","commonSituations":"Offset checkpoint location on S3/HDFS with permission or network problems; the checkpoint path deleted or bucket unavailable while a Spark structured-streaming query starts.","solutions":["Check the wrapped IOException cause for filesystem/permission details on initialOffsetLocation.","Verify the checkpoint location is writable with the running job's credentials.","Confirm storage availability/quotas, then restart the streaming query."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// ensure checkpoint location is creatable/writable before starting the stream\nFileIO io = table.io();\nio.checkOutputFile(initialOffsetLocation); // throws if unwritable","typeGuard":null,"tryCatchPattern":"try { stream.initialOffset(); } catch (UncheckedIOException e) { log.error(\"offset write failed at {}: {}\", loc, e.getCause()); throw e; }","preventionTips":["Verify write permissions and credentials on the streaming checkpoint location","Keep checkpoint location on reliable storage with no lifecycle deletion rules","Pre-create/validate the checkpoint path before submitting the query"],"tags":["spark","streaming","io","offset-checkpoint"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}