What causes "Multi-part upload with id '{uploadId}' to {key}" in apache/hadoop?

Inspect the nested cause of the IOException — it contains the real OSS SDK error (e.g., InvalidAccessKeyId, RequestTimeTooSkewed, SocketTimeout) and fix that

apache/hadoop · critical · IOException

Multi-part upload with id '{uploadId}' to {key}

Error message

Multi-part upload with id '{uploadId}' to {key}

What it means

Thrown by AliyunOSSBlockOutputStream.waitForAllPartUploads() when one of the ListenableFuture<PartETag> part uploads failed with an ExecutionException. On this path the code cancels all remaining part futures, calls store.abortMultipartUpload(key, uploadId) so OSS discards the incomplete parts, and wraps the cause in an IOException carrying the uploadId and destination key.

Source

Thrown at hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSBlockOutputStream.java:273

  private List<PartETag> waitForAllPartUploads() throws IOException {
    LOG.debug("Waiting for {} uploads to complete", partETagsFutures.size());
    try {
      return Futures.allAsList(partETagsFutures).get();
    } catch (InterruptedException ie) {
      LOG.warn("Interrupted partUpload", ie);
      Thread.currentThread().interrupt();
      return null;
    } catch (ExecutionException ee) {
      //there is no way of recovering so abort
      //cancel all partUploads
      LOG.debug("While waiting for upload completion", ee);
      LOG.debug("Cancelling futures");
      for (ListenableFuture<PartETag> future : partETagsFutures) {
        future.cancel(true);
      }
      //abort multipartupload
      store.abortMultipartUpload(key, uploadId);
      throw new IOException("Multi-part upload with id '" + uploadId
        + "' to " + key, ee);
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Inspect the nested cause of the IOException — it contains the real OSS SDK error (e.g., InvalidAccessKeyId, RequestTimeTooSkewed, SocketTimeout) and fix that
For long writes with temporary credentials, refresh STS tokens with a longer validity or use a credentials provider that auto-refreshes
Retry the write at the job/task level; the upload was aborted so no partial object remains
For throttling, increase fs.oss.multipart.size so fewer part requests are issued, or reduce concurrent writers

Defensive patterns

Strategy: retry

Try / catch

catch (IOException e) { Throwable cause = e.getCause(); if (cause instanceof OSSException) { String code = ((OSSException) cause).getErrorCode(); /* handle InvalidAccessKeyId / RequestTimeTooSkewed / SlowDown */ } throw e; }

Prevention

Use STS tokens with validity longer than the maximum expected write duration, or auto-refreshing providers
Retry the write at the job/task level — the connector already aborted the multipart upload
Inspect the nested cause before changing config; it carries the real OSS error code

When it happens

Trigger: Any individual part PUT to OSS failing: 403 signature/permission errors, expired STS token mid-upload, network reset, bucket-level throttling (RequestTimeTooSkewed, SlowDown), or the OSS client throwing inside the executor. The exception's cause holds the actual OSS error.

Common situations: STS security token expiring during a long write of a large file; transient network issues between the cluster and the OSS endpoint; RAM policy missing oss:PutObject on the bucket; throttling under heavy parallel part uploads (fs.oss.multipart.size too small, many concurrent tasks).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ca2db2751351a3d8. Report an issue: GitHub.