{"record":{"id":"eae94f54475ae78b","repo":"apache/iceberg","slug":"s3ablockoutputstream-failed-to-upload-object-after","errorCode":null,"errorMessage":"S3ABlockOutputStream failed to upload object after stream was closed","messagePattern":"S3ABlockOutputStream failed to upload object after stream was closed","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java","lineNumber":193,"sourceCode":"\n    @Override\n    public void flush() throws IOException {\n      stream.flush();\n    }\n\n    @Override\n    public void close() throws IOException {\n      stream.close();\n      this.closed = true;\n      // {@link org.apache.hadoop.fs.s3a.S3ABlockOutputStream#close()} calls {@link\n      // org.apache.hadoop.fs.s3a.S3ABlockOutputStream#putObject()}\n      // which doesn't throw an exception when interrupted.\n      // Need to check the interrupted flag to detect failed object upload\n      // and propagate the error up.\n      if (Thread.interrupted()\n          && \"org.apache.hadoop.fs.s3a.S3ABlockOutputStream\"\n              .equals(stream.getWrappedStream().getClass().getName())) {\n        throw new IOException(\n            \"S3ABlockOutputStream failed to upload object after stream was closed\");\n      }\n    }\n\n    @SuppressWarnings({\"checkstyle:NoFinalizer\", \"Finalize\", \"deprecation\"})\n    @Override\n    protected void finalize() throws Throwable {\n      super.finalize();\n      if (!closed) {\n        close(); // releasing resources is more important than printing the warning\n        String trace =\n            Joiner.on(\"\\n\\t\").join(Arrays.copyOfRange(createStack, 1, createStack.length));\n        LOG.warn(\"Unclosed output stream created by:\\n\\t{}\", trace);\n      }\n    }\n  }\n\n  private static class WrappedSeekableInputStream extends FSInputStream","sourceCodeStart":175,"sourceCodeEnd":211,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java#L175-L211","documentation":"HadoopStreams' closeable wrapper detects a Hadoop bug where interrupting a thread closing an S3ABlockOutputStream swallows the upload failure. If the thread's interrupt flag is set after close and the wrapped stream is an S3ABlockOutputStream, it rethrows an IOException so the failed S3 upload is not silently treated as success.","triggerScenarios":"Closing an S3A-backed output stream on a thread that was interrupted during the close/multipart-upload; the S3A multi-part upload actually failed but Hadoop's close() returned normally.","commonSituations":"Task cancellation/timeouts in Spark/Flink interrupting writer threads while committing S3A uploads; executor shutdown interrupting long multipart uploads; S3 throttling or network failure during the final upload combined with cancellation.","solutions":["Don't interrupt threads performing S3A writes; allow close()/commit to complete before cancelling","Ensure the underlying S3 upload succeeds (check S3 bucket policies, networking, endpoint health) and retry the write on failure","Upgrade Hadoop (s3a) to a version fixing the swallow-on-interrupt behavior in S3ABlockOutputStream","Treat this exception as a real failed upload: rewrite the object rather than assuming it was persisted"],"exampleFix":"// before\nstream.close(); // assumed success even when interrupted\n// after\ntry { stream.close(); }\ncatch (IOException e) {\n  if (e.getMessage().contains(\"failed to upload\")) {\n    LOG.error(\"S3A upload did not complete; object must be rewritten\", e);\n    throw e; // do not commit this file\n  }\n  throw e;\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":"boolean isS3ABlockStream = stream instanceof org.apache.hadoop.fs.FSDataOutputStream\n    || \"org.apache.hadoop.fs.s3a.S3ABlockOutputStream\".equals(stream.getClass().getName());","tryCatchPattern":"try { stream.close(); }\ncatch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"S3ABlockOutputStream failed to upload\")) {\n    LOG.error(\"S3A upload failed after close; object is NOT committed\", e);\n  }\n  throw e; // never treat close as successful when this fires\n}","preventionTips":["Avoid interrupting threads while S3A output streams are open/closing","Complete writer close before task cancellation deadlines","Verify uploaded objects exist (length > 0) before committing metadata","Upgrade Hadoop s3a client to get interrupt-handling fixes"],"tags":["s3","hadoop","upload","interrupt"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}