{"record":{"id":"567d2cb21d383784","repo":"apache/hadoop","slug":"number-of-parts-in-multipart-upload-exceeded-curr","errorCode":null,"errorMessage":"Number of parts in multipart upload exceeded. Current part count = %s, Part count limit = %s ","messagePattern":"Number of parts in multipart upload exceeded\\. Current part count = (.+?), Part count limit = (.+?) ","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java","lineNumber":667,"sourceCode":"  @Override\n  public UploadPartRequest.Builder newUploadPartRequestBuilder(\n      String destKey,\n      String uploadId,\n      int partNumber,\n      boolean isLastPart,\n      long size) throws PathIOException {\n    checkNotNull(uploadId);\n    checkArgument(size >= 0, \"Invalid partition size %s\", size);\n    checkArgument(partNumber > 0,\n        \"partNumber must be between 1 and %s inclusive, but is %s\",\n        multipartPartCountLimit, partNumber);\n\n    LOG.debug(\"Creating part upload request for {} #{} size {}\",\n        uploadId, partNumber, size);\n    final String pathErrorMsg = \"Number of parts in multipart upload exceeded.\"\n        + \" Current part count = %s, Part count limit = %s \";\n    if (partNumber > multipartPartCountLimit) {\n      throw new PathIOException(destKey,\n          String.format(pathErrorMsg, partNumber, multipartPartCountLimit));\n    }\n    UploadPartRequest.Builder builder = UploadPartRequest.builder()\n        .bucket(getBucket())\n        .key(destKey)\n        .uploadId(uploadId)\n        .partNumber(partNumber)\n        .contentLength(size);\n    if (isLastPart) {\n      builder.sdkPartType(SdkPartType.LAST);\n    }\n    uploadPartEncryptionParameters(builder);\n\n    // Set the request timeout for the part upload\n    setRequestTimeout(builder, partUploadTimeout);\n\n    if (checksumAlgorithm != null) {\n      builder.checksumAlgorithm(checksumAlgorithm);","sourceCodeStart":649,"sourceCodeEnd":685,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java#L649-L685","documentation":"Thrown by RequestFactoryImpl (hadoop-aws) when a multipart upload would need a part number above the configured part count limit (default 10000, matching the hard Amazon S3 limit of 10,000 parts per multipart upload). The limit is normally DEFAULT_UPLOAD_PART_COUNT_LIMIT=10000 and is only overridable via the test-only key fs.s3a.internal.upload.part.count.limit. The error means the file being written or copied exceeds partSize * partCountLimit bytes with the current fs.s3a.multipart.size (default 64 MB -> ~640 TB ceiling).","triggerScenarios":"Writing or copying an S3A file whose size exceeds fs.s3a.multipart.size * 10000 (e.g. 64 MB parts cap at 640 TB; a misconfigured 5 MB part size caps at ~50 GB); large distcp copy or rename of a huge object; any S3AFileSystem.create()/copyFromLocalFile() of an oversized file with too-small part size; a test forcing fs.s3a.internal.upload.part.count.limit below the required part count.","commonSituations":"Operators lowering fs.s3a.multipart.size to reduce memory or 'optimize' small-file writes, then hitting the ceiling on very large objects; distcp jobs moving multi-terabyte objects; upgrading tooling that writes multi-TB archives to s3a://. Rarely seen with defaults because 64 MB parts allow 640 TB files.","solutions":["Increase fs.s3a.multipart.size so that partSize * 10000 exceeds the largest object you write (e.g. set fs.s3a.multipart.size to 536870912 (512 MB) or 1073741824 (1 GB); S3 allows up to 5 GB per part).","If the value of fs.s3a.multipart.size was lowered (check conf.getPropertySources for the offending file), remove or raise that override.","For gigantic single objects, split the output into multiple files at the application level so each stays under partSize * 10000.","Never rely on fs.s3a.internal.upload.part.count.limit - it is @VisibleForTesting and does not lift the real S3 10,000-part service limit."],"exampleFix":"// before: default 64MB parts cap uploads at 64MB * 10000 = 640TB,\n// a lowered value caps much lower\n<property><name>fs.s3a.multipart.size</name><value>5242880</value></property>\n\n// after: 512MB parts support objects up to 5TB (S3's max object size)\n<property><name>fs.s3a.multipart.size</name><value>536870912</value></property>","handlingStrategy":"validation","validationCode":"// before uploading, confirm size fits within parts * limit\nlong partSize = conf.getLong(\"fs.s3a.multipart.size\", 67108864L);\nlong maxObjectSize = partSize * 10_000L; // S3 hard part limit\nif (srcLen > maxObjectSize) {\n  throw new IOException(\"Object of \" + srcLen + \" bytes exceeds \"\n      + maxObjectSize + \"; raise fs.s3a.multipart.size\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  fs.copyFromLocalFile(src, dst);\n} catch (PathIOException e) {\n  if (e.getMessage().contains(\"part count limit\")) {\n    // raise fs.s3a.multipart.size and retry once\n  } else { throw e; }\n}","preventionTips":["Size fs.s3a.multipart.size so partSize * 10000 > your largest object (512 MB parts cover S3's 5 TB max).","Never lower fs.s3a.multipart.size for large-file workloads without recomputing the size ceiling.","Check the effective value with 'hadoop conf' after config changes across all nodes and job submissions."],"tags":["s3","hadoop-aws","multipart-upload","configuration","file-size-limit"],"backgroundTag":"s3-multipart-part-count-limit","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}