apache/iceberg · critical · IOException

S3ABlockOutputStream failed to upload object after stream…

Error message

S3ABlockOutputStream failed to upload object after stream was closed

What it means

HadoopStreams' closeable wrapper detects a Hadoop bug where interrupting a thread closing an S3ABlockOutputStream swallows the upload failure. If the thread's interrupt flag is set after close and the wrapped stream is an S3ABlockOutputStream, it rethrows an IOException so the failed S3 upload is not silently treated as success.

Solutions

Don't interrupt threads performing S3A writes; allow close()/commit to complete before cancelling
Ensure the underlying S3 upload succeeds (check S3 bucket policies, networking, endpoint health) and retry the write on failure
Upgrade Hadoop (s3a) to a version fixing the swallow-on-interrupt behavior in S3ABlockOutputStream
Treat this exception as a real failed upload: rewrite the object rather than assuming it was persisted

Example fix

// before
stream.close(); // assumed success even when interrupted
// after
try { stream.close(); }
catch (IOException e) {
  if (e.getMessage().contains("failed to upload")) {
    LOG.error("S3A upload did not complete; object must be rewritten", e);
    throw e; // do not commit this file
  }
  throw e;
}

Defensive patterns

Strategy: try-catch

Type guard

boolean isS3ABlockStream = stream instanceof org.apache.hadoop.fs.FSDataOutputStream
    || "org.apache.hadoop.fs.s3a.S3ABlockOutputStream".equals(stream.getClass().getName());

Try / catch

try { stream.close(); }
catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("S3ABlockOutputStream failed to upload")) {
    LOG.error("S3A upload failed after close; object is NOT committed", e);
  }
  throw e; // never treat close as successful when this fires
}

Prevention

Avoid interrupting threads while S3A output streams are open/closing
Complete writer close before task cancellation deadlines
Verify uploaded objects exist (length > 0) before committing metadata
Upgrade Hadoop s3a client to get interrupt-handling fixes

When it happens

Trigger: Closing an S3A-backed output stream on a thread that was interrupted during the close/multipart-upload; the S3A multi-part upload actually failed but Hadoop's close() returned normally.

Common situations: Task cancellation/timeouts in Spark/Flink interrupting writer threads while committing S3A uploads; executor shutdown interrupting long multipart uploads; S3 throttling or network failure during the final upload combined with cancellation.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/eae94f54475ae78b. Report an issue: GitHub.

Appendix: source

Thrown at core/src/main/java/org/apache/iceberg/hadoop/HadoopStreams.java:193

    @Override
    public void flush() throws IOException {
      stream.flush();
    }

    @Override
    public void close() throws IOException {
      stream.close();
      this.closed = true;
      // {@link org.apache.hadoop.fs.s3a.S3ABlockOutputStream#close()} calls {@link
      // org.apache.hadoop.fs.s3a.S3ABlockOutputStream#putObject()}
      // which doesn't throw an exception when interrupted.
      // Need to check the interrupted flag to detect failed object upload
      // and propagate the error up.
      if (Thread.interrupted()
          && "org.apache.hadoop.fs.s3a.S3ABlockOutputStream"
              .equals(stream.getWrappedStream().getClass().getName())) {
        throw new IOException(
            "S3ABlockOutputStream failed to upload object after stream was closed");
      }
    }

    @SuppressWarnings({"checkstyle:NoFinalizer", "Finalize", "deprecation"})
    @Override
    protected void finalize() throws Throwable {
      super.finalize();
      if (!closed) {
        close(); // releasing resources is more important than printing the warning
        String trace =
            Joiner.on("\n\t").join(Arrays.copyOfRange(createStack, 1, createStack.length));
        LOG.warn("Unclosed output stream created by:\n\t{}", trace);
      }
    }
  }

  private static class WrappedSeekableInputStream extends FSInputStream

View on GitHub (pinned to 86d9c8fc54)