{"record":{"id":"787196223c6515dc","repo":"apache/hadoop","slug":"file-to-upload-size-d-is-too-big-to-be-uploaded","errorCode":null,"errorMessage":"File to upload (size %d) is too big to be uploaded in parts of size %d","messagePattern":"File to upload \\(size (.+?)\\) is too big to be uploaded in parts of size (.+?)","errorType":"exception","errorClass":"PathIOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitOperations.java","lineNumber":573,"sourceCode":"      commitData.setDestinationKey(destKey);\n      commitData.setBucket(fs.getBucket());\n      commitData.touch(System.currentTimeMillis());\n      commitData.setUploadId(uploadId);\n      commitData.setUri(destURI);\n      commitData.setText(partition != null ? \"partition: \" + partition : \"\");\n      commitData.setLength(length);\n\n      long numParts = (length / uploadPartSize +\n          ((length % uploadPartSize) > 0 ? 1 : 0));\n      // always write one part, even if it is just an empty one\n      if (numParts == 0) {\n        numParts = 1;\n      }\n      if (numParts > InternalConstants.DEFAULT_UPLOAD_PART_COUNT_LIMIT) {\n        // fail if the file is too big.\n        // it would be possible to be clever here and recalculate the part size,\n        // but this is not currently done.\n        throw new PathIOException(destPath.toString(),\n            String.format(\"File to upload (size %d)\"\n                + \" is too big to be uploaded in parts of size %d\",\n                numParts, length));\n      }\n\n      final int partCount = (int) numParts;\n      LOG.debug(\"File size is {}, number of parts to upload = {}\",\n          length, partCount);\n\n      // Open the file to upload.\n      List<CompletedPart> parts = uploadFileData(\n          uploadId,\n          localFile,\n          destKey,\n          progress,\n          length,\n          partCount,\n          uploadPartSize);","sourceCodeStart":555,"sourceCodeEnd":591,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/impl/CommitOperations.java#L555-L591","documentation":"PathIOException from CommitOperations.uploadFileToPendingCommit when the number of computed multipart parts exceeds InternalConstants.DEFAULT_UPLOAD_PART_COUNT_LIMIT (10000), the S3 limit on parts per multipart upload. The code deliberately fails instead of silently adjusting the part size (the comment says being 'clever' here is not currently done). Warning: the format arguments are swapped relative to the text -- the number printed after 'size' is actually numParts and the number printed after 'parts of size' is the file length, so read the message accordingly.","triggerScenarios":"uploadFileToPendingCommit on a local file whose length / uploadPartSize (derived from fs.s3a.multipart.size) rounds up to more than 10000 parts; with the default 5 MB part window any single output file over roughly 50 GB hits it, and larger part-size configs raise the ceiling proportionally.","commonSituations":"A single reducer or Spark task emits one enormous output file (skewed data, too few partitions); fs.s3a.multipart.size was lowered for small-file reasons and now starves big files.","solutions":["Raise fs.s3a.multipart.size so that fileLength / partSize stays at or below 10000 (partSize >= ceil(maxFileLength / 10000))","Reduce the maximum per-task output size: increase reduce tasks / spark.sql.shuffle.partitions or split the writing job so no single file crosses partSize * 10000","When triaging, remember the two numbers in the message are swapped: first = part count, second = file length in bytes"],"exampleFix":"<!-- before: small part size caps files at ~5MB*10000 -->\n<property><name>fs.s3a.multipart.size</name><value>5M</value></property>\n\n<!-- after: 100M parts allow single files up to ~1TB -->\n<property><name>fs.s3a.multipart.size</name><value>100M</value></property>","handlingStrategy":"validation","validationCode":"long maxFile = (long) partSize * InternalConstants.DEFAULT_UPLOAD_PART_COUNT_LIMIT; // partSize * 10000\nif (localFile.length() > maxFile) {\n  throw new IOException(\"File \" + localFile + \" exceeds upload budget of \"\n      + maxFile + \" bytes with part size \" + partSize\n      + \" -- raise fs.s3a.multipart.size or split the output\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  ops.uploadFileToPendingCommit(localFile, destPath, partition, partSize, progress);\n} catch (PathIOException e) {\n  // remember: the two numbers in 'too big to be uploaded' are swapped (numParts, length)\n  throw new IOException(\"Output too large for part size; increase fs.s3a.multipart.size\", e);\n}","preventionTips":["Size fs.s3a.multipart.size against your largest single task output, not your average file","Watch for data skew that funnels a whole partition into one task/file","Add a precommit length check when output sizes are unpredictable"],"tags":["s3a","multipart-upload","file-size","committer"],"backgroundTag":"file-too-large","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}