apache/hadoop · error · PathIOException

File to upload (size %d) is too big to be uploaded in parts

Error message

File to upload (size %d) is too big to be uploaded in parts of size %d

What it means

PathIOException from CommitOperations.uploadFileToPendingCommit when the number of computed multipart parts exceeds InternalConstants.DEFAULT_UPLOAD_PART_COUNT_LIMIT (10000), the S3 limit on parts per multipart upload. The code deliberately fails instead of silently adjusting the part size (the comment says being 'clever' here is not currently done). Warning: the format arguments are swapped relative to the text -- the number printed after 'size' is actually numParts and the number printed after 'parts of size' is the file length, so read the message accordingly.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitOperations.java:573

      commitData.setDestinationKey(destKey);
      commitData.setBucket(fs.getBucket());
      commitData.touch(System.currentTimeMillis());
      commitData.setUploadId(uploadId);
      commitData.setUri(destURI);
      commitData.setText(partition != null ? "partition: " + partition : "");
      commitData.setLength(length);

      long numParts = (length / uploadPartSize +
          ((length % uploadPartSize) > 0 ? 1 : 0));
      // always write one part, even if it is just an empty one
      if (numParts == 0) {
        numParts = 1;
      }
      if (numParts > InternalConstants.DEFAULT_UPLOAD_PART_COUNT_LIMIT) {
        // fail if the file is too big.
        // it would be possible to be clever here and recalculate the part size,
        // but this is not currently done.
        throw new PathIOException(destPath.toString(),
            String.format("File to upload (size %d)"
                + " is too big to be uploaded in parts of size %d",
                numParts, length));
      }

      final int partCount = (int) numParts;
      LOG.debug("File size is {}, number of parts to upload = {}",
          length, partCount);

      // Open the file to upload.
      List<CompletedPart> parts = uploadFileData(
          uploadId,
          localFile,
          destKey,
          progress,
          length,
          partCount,
          uploadPartSize);

View on GitHub (pinned to 2add963021)

Solutions

Raise fs.s3a.multipart.size so that fileLength / partSize stays at or below 10000 (partSize >= ceil(maxFileLength / 10000))
Reduce the maximum per-task output size: increase reduce tasks / spark.sql.shuffle.partitions or split the writing job so no single file crosses partSize * 10000
When triaging, remember the two numbers in the message are swapped: first = part count, second = file length in bytes

Example fix

<!-- before: small part size caps files at ~5MB*10000 -->
<property><name>fs.s3a.multipart.size</name><value>5M</value></property>

<!-- after: 100M parts allow single files up to ~1TB -->
<property><name>fs.s3a.multipart.size</name><value>100M</value></property>

Defensive patterns

Strategy: validation

Validate before calling

long maxFile = (long) partSize * InternalConstants.DEFAULT_UPLOAD_PART_COUNT_LIMIT; // partSize * 10000
if (localFile.length() > maxFile) {
  throw new IOException("File " + localFile + " exceeds upload budget of "
      + maxFile + " bytes with part size " + partSize
      + " -- raise fs.s3a.multipart.size or split the output");
}

Try / catch

try {
  ops.uploadFileToPendingCommit(localFile, destPath, partition, partSize, progress);
} catch (PathIOException e) {
  // remember: the two numbers in 'too big to be uploaded' are swapped (numParts, length)
  throw new IOException("Output too large for part size; increase fs.s3a.multipart.size", e);
}

Prevention

Size fs.s3a.multipart.size against your largest single task output, not your average file
Watch for data skew that funnels a whole partition into one task/file
Add a precommit length check when output sizes are unpredictable

When it happens

Trigger: uploadFileToPendingCommit on a local file whose length / uploadPartSize (derived from fs.s3a.multipart.size) rounds up to more than 10000 parts; with the default 5 MB part window any single output file over roughly 50 GB hits it, and larger part-size configs raise the ceiling proportionally.

Common situations: A single reducer or Spark task emits one enormous output file (skewed data, too few partitions); fs.s3a.multipart.size was lowered for small-file reasons and now starves big files.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/787196223c6515dc. Report an issue: GitHub.