apache/hadoop · error · PathIOException

Number of parts in multipart upload exceeded. Current part c

Error message

Number of parts in multipart upload exceeded. Current part count = %s, Part count limit = %s 

What it means

Thrown by RequestFactoryImpl (hadoop-aws) when a multipart upload would need a part number above the configured part count limit (default 10000, matching the hard Amazon S3 limit of 10,000 parts per multipart upload). The limit is normally DEFAULT_UPLOAD_PART_COUNT_LIMIT=10000 and is only overridable via the test-only key fs.s3a.internal.upload.part.count.limit. The error means the file being written or copied exceeds partSize * partCountLimit bytes with the current fs.s3a.multipart.size (default 64 MB -> ~640 TB ceiling).

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java:667

  @Override
  public UploadPartRequest.Builder newUploadPartRequestBuilder(
      String destKey,
      String uploadId,
      int partNumber,
      boolean isLastPart,
      long size) throws PathIOException {
    checkNotNull(uploadId);
    checkArgument(size >= 0, "Invalid partition size %s", size);
    checkArgument(partNumber > 0,
        "partNumber must be between 1 and %s inclusive, but is %s",
        multipartPartCountLimit, partNumber);

    LOG.debug("Creating part upload request for {} #{} size {}",
        uploadId, partNumber, size);
    final String pathErrorMsg = "Number of parts in multipart upload exceeded."
        + " Current part count = %s, Part count limit = %s ";
    if (partNumber > multipartPartCountLimit) {
      throw new PathIOException(destKey,
          String.format(pathErrorMsg, partNumber, multipartPartCountLimit));
    }
    UploadPartRequest.Builder builder = UploadPartRequest.builder()
        .bucket(getBucket())
        .key(destKey)
        .uploadId(uploadId)
        .partNumber(partNumber)
        .contentLength(size);
    if (isLastPart) {
      builder.sdkPartType(SdkPartType.LAST);
    }
    uploadPartEncryptionParameters(builder);

    // Set the request timeout for the part upload
    setRequestTimeout(builder, partUploadTimeout);

    if (checksumAlgorithm != null) {
      builder.checksumAlgorithm(checksumAlgorithm);

View on GitHub (pinned to 2add963021)

Solutions

1. Increase fs.s3a.multipart.size so that partSize * 10000 exceeds the largest object you write (e.g. set fs.s3a.multipart.size to 536870912 (512 MB) or 1073741824 (1 GB); S3 allows up to 5 GB per part).
2. If the value of fs.s3a.multipart.size was lowered (check conf.getPropertySources for the offending file), remove or raise that override.
3. For gigantic single objects, split the output into multiple files at the application level so each stays under partSize * 10000.
4. Never rely on fs.s3a.internal.upload.part.count.limit - it is @VisibleForTesting and does not lift the real S3 10,000-part service limit.

Example fix

// before: default 64MB parts cap uploads at 64MB * 10000 = 640TB,
// a lowered value caps much lower
<property><name>fs.s3a.multipart.size</name><value>5242880</value></property>

// after: 512MB parts support objects up to 5TB (S3's max object size)
<property><name>fs.s3a.multipart.size</name><value>536870912</value></property>

// before uploading, confirm size fits within parts * limit
long partSize = conf.getLong("fs.s3a.multipart.size", 67108864L);
long maxObjectSize = partSize * 10_000L; // S3 hard part limit
if (srcLen > maxObjectSize) {
  throw new IOException("Object of " + srcLen + " bytes exceeds "
      + maxObjectSize + "; raise fs.s3a.multipart.size");
}

try {
  fs.copyFromLocalFile(src, dst);
} catch (PathIOException e) {
  if (e.getMessage().contains("part count limit")) {
    // raise fs.s3a.multipart.size and retry once
  } else { throw e; }
}

When it happens

Trigger: Writing or copying an S3A file whose size exceeds fs.s3a.multipart.size * 10000 (e.g. 64 MB parts cap at 640 TB; a misconfigured 5 MB part size caps at ~50 GB); large distcp copy or rename of a huge object; any S3AFileSystem.create()/copyFromLocalFile() of an oversized file with too-small part size; a test forcing fs.s3a.internal.upload.part.count.limit below the required part count.

Common situations: Operators lowering fs.s3a.multipart.size to reduce memory or 'optimize' small-file writes, then hitting the ceiling on very large objects; distcp jobs moving multi-terabyte objects; upgrading tooling that writes multi-TB archives to s3a://. Rarely seen with defaults because 64 MB parts allow 640 TB files.

Related errors

• Unable to create OutputStream with the given multipart uploa
• Multipart IO request {sdkRequest} rejected {header}
• Bucket does not exist. Accessing with fs.s3a.endpoint set t
• Cannot find password option {key}
• SSE-C is enabled but no encryption key was declared in fs.s3

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/567d2cb21d383784. Report an issue: GitHub.