apache/hadoop · error · RuntimeException

parts length mismatched: %d != %d

Error message

parts length mismatched: %d != %d

What it means

FileStore.completeUpload() counts the part files present in the upload's staging directory and compares against the uploadParts list handed in; if the counts differ it throws 'parts length mismatched'. The caller's part list and the parts actually staged on disk no longer describe the same upload — parts are missing from the list, or the directory holds extra/stale part files.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java:414

    }

    if (!tmpFile.renameTo(partFile)) {
      throw new RuntimeException("failed to put file since rename fail.");
    }
  }

  @Override
  public byte[] completeUpload(String key, String uploadId, List<Part> uploadParts) {
    Preconditions.checkArgument(uploadParts != null && uploadParts.size() > 0,
        "upload parts cannot be null or empty.");
    File uploadDir = uploadPath(key, uploadId).toFile();
    if (!uploadDir.exists()) {
      throw new RuntimeException("cannot locate the upload id: " + uploadId);
    }

    List<Integer> partNums = listPartNums(uploadDir);
    if (partNums.size() != uploadParts.size()) {
      throw new RuntimeException(String.format("parts length mismatched: %d != %d",
          partNums.size(), uploadParts.size()));
    }

    Collections.sort(partNums);
    uploadParts.sort(Comparator.comparingInt(Part::num));

    Path keyPath = path(encode(key));
    File tmpFile = createTmpFile(keyPath.toFile());
    try (FileOutputStream outputStream = new FileOutputStream(tmpFile);
        FileChannel outputChannel = outputStream.getChannel()) {
      int offset = 0;
      for (int i = 0; i < partNums.size(); i++) {
        Part part = uploadParts.get(i);
        if (part.num() != partNums.get(i)) {
          throw new RuntimeException(
              String.format("part num mismatched: %d != %d", part.num(), partNums.get(i)));
        }

View on GitHub (pinned to 2add963021)

Solutions

  1. Build the parts list exclusively from the Part objects returned by uploadPart()/uploadPartCopy() and pass that exact list to completeUpload
  2. Ensure every part upload succeeded (collect and check results) before completing
  3. Never modify or prune files inside __STAGING__/<key>/<uploadId> by hand; if the dir is suspect, abort the upload and start a new one
  4. Give each concurrent writer its own uploadId via createMultipartUpload instead of sharing one

Example fix

// before: hand-built parts list that lost part 1
List<Part> parts = List.of(new Part(2, len2, etag2)); // parts length mismatched: 1 != 2
// after: collect exactly what uploadPart returned, in upload order
List<Part> parts = new ArrayList<>();
for (int n = 1; n <= totalParts; n++) {
  parts.add(storage.uploadPart(key, uploadId, n, providerFor(n), lenFor(n)));
}
storage.completeUpload(key, uploadId, parts);
Defensive patterns

Strategy: validation

Validate before calling

List<Part> parts = new ArrayList<>();
for (int n = 1; n <= totalParts; n++) {
  Part p = storage.uploadPart(key, uploadId, n, providerFor(n), lenFor(n));
  if (p == null) throw new IOException("part " + n + " not uploaded");
  parts.add(p);
}
if (parts.size() != totalParts) {
  throw new IOException("staged " + parts.size() + " parts, expected " + totalParts);
}
storage.completeUpload(key, uploadId, parts);

Prevention

When it happens

Trigger: completeUpload where some uploadPart call never ran or failed silently, a part file was deleted from the staging dir, stale part files from an earlier numbering scheme linger, or the caller hand-built the parts list instead of using the Part objects returned by uploadPart().

Common situations: Loop that uploads parts but drops one on an exception path; retry frameworks that re-run a subset of parts; tests listing part files manually; concurrent writers sharing an uploadId.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bc94ac7ac02f4c76. Report an issue: GitHub.