{"record":{"id":"4c31d90b2b615cb9","repo":"apache/iceberg","slug":"failed-writing-offset-to-s","errorCode":null,"errorMessage":"Failed writing offset to: %s","messagePattern":"Failed writing offset to: (.+?)","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"critical","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java","lineNumber":301,"sourceCode":"      }\n\n      table.refresh();\n      StreamingOffset offset = MicroBatchUtils.determineStartingOffset(table, fromTimestamp);\n\n      OutputFile outputFile = io.newOutputFile(initialOffsetLocation);\n      writeOffset(offset, outputFile);\n\n      return offset;\n    }\n\n    private void writeOffset(StreamingOffset offset, OutputFile file) {\n      try (OutputStream outputStream = file.create()) {\n        BufferedWriter writer =\n            new BufferedWriter(new OutputStreamWriter(outputStream, StandardCharsets.UTF_8));\n        writer.write(offset.json());\n        writer.flush();\n      } catch (IOException ioException) {\n        throw new UncheckedIOException(\n            String.format(\"Failed writing offset to: %s\", initialOffsetLocation), ioException);\n      }\n    }\n\n    private StreamingOffset readOffset(InputFile file) {\n      try (InputStream in = file.newStream()) {\n        return StreamingOffset.fromJson(in);\n      } catch (IOException ioException) {\n        throw new UncheckedIOException(\n            String.format(\"Failed reading offset from: %s\", initialOffsetLocation), ioException);\n      }\n    }\n  }\n}\n","sourceCodeStart":283,"sourceCodeEnd":316,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java#L283-L316","documentation":"SparkMicroBatchStream writes the current StreamingOffset as JSON to a file under the initial offset location so Spark Structured Streaming can track scan progress. This UncheckedIOException wraps any IOException that occurs while creating or writing that offset file. It indicates the streaming query cannot persist its checkpoint offset, so the batch cannot proceed safely.","triggerScenarios":"initialOffset() calls writeOffset and the underlying FileSystem (file.create()) fails to create or write the offset file — e.g. permission denied on the checkpoint/offset directory, HDFS/S3 transient outage, disk full, or the offset location was deleted between checks.","commonSituations":"S3/HDFS transient unavailability during a streaming query start; checkpoint directory permissions changed or the offset file made read-only; container disk pressure; offset location pointing to a wrong or removed path after a config change.","solutions":["Check that the offset/checkpoint location exists and is writable by the job's identity (fs permissions, IAM/S3 bucket policy).","Inspect the wrapped IOException (cause) for the root filesystem error and fix it (disk full, connectivity, throttling).","Retry the streaming query after resolving transient storage outages; clear a corrupt/partial offset file only after verifying checkpoint semantics.","Pin/stabilize the underlying filesystem (Hadoop aws/hdfs versions, retry settings) so file.create() is resilient."],"exampleFix":"// before: raw UncheckedIOException surfaces from stream init\nStreamingOffset offset = stream.initialOffset();\n// after: pre-validate offset location writability\nPath offsetPath = new Path(initialOffsetLocation);\nFileSystem fs = offsetPath.getFileSystem(conf);\nPreconditions.checkState(fs.mkdirs(offsetPath.getParent()), \"Offset dir not creatable: %s\", offsetPath.getParent());\nStreamingOffset offset = stream.initialOffset();","handlingStrategy":"validation","validationCode":"Path offsetPath = new Path(initialOffsetLocation);\nFileSystem fs = offsetPath.getFileSystem(conf);\nif (!fs.exists(offsetPath.getParent())) {\n  fs.mkdirs(offsetPath.getParent());\n}\nFileStatus st = fs.getFileStatus(offsetPath.getParent());\n// verify write access before starting the stream\n","typeGuard":null,"tryCatchPattern":"try { offset = stream.initialOffset(); } catch (UncheckedIOException e) {\n  LOG.error(\"Offset write failed at {}: {}\", initialOffsetLocation, e.getCause());\n  throw new IllegalStateException(\"Fix offset location access/retry storage\", e);\n}","preventionTips":["Pre-create and verify permissions on the checkpoint/offset directory before starting the query","Do not share checkpoint locations between queries or clean them while running","Monitor disk/HDFS/S3 health and configure client retries"],"tags":["spark","structured-streaming","io","checkpoint"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}