apache/hadoop · error · RuntimeException

Upload directory %s already exits

Error message

Upload directory %s already exits

What it means

FileStore.uploadPartCopy() throws 'Upload directory %s already exits' — note the message is inverted (and misspelled): the code checks `if (!uploadDir.exists())` and throws, so the exception actually means the staging directory for <root>/__STAGING__/<encodedDstKey>/<uploadId> is MISSING, not already present. The uploadId is unknown for this destination key, most often because the multipart upload was created for a different key string or never created at all.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java:542

                      id.toFile().getName(), MIN_PART_SIZE, MAX_PART_COUNT));
            } catch (IOException e) {
              throw new RuntimeException(e);
            }
          })
          .sorted()
          .collect(Collectors.toList());
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

  @Override
  public Part uploadPartCopy(
      String srcKey, String dstKey, String uploadId, int partNum, long copySourceRangeStart,
      long copySourceRangeEnd) {
    File uploadDir = uploadPath(dstKey, uploadId).toFile();
    if (!uploadDir.exists()) {
      throw new RuntimeException(String.format("Upload directory %s already exits", uploadDir));
    }
    File partFile = new File(uploadDir, String.valueOf(partNum));
    int fileSize = (int) (copySourceRangeEnd - copySourceRangeStart + 1);
    try (InputStream is = get(srcKey, copySourceRangeStart, fileSize).stream();
        FileOutputStream fos = new FileOutputStream(partFile)) {
      byte[] data = new byte[fileSize];
      IOUtils.readFully(is, data);
      fos.write(data);
      return new Part(partNum, fileSize, DigestUtils.md5Hex(data));
    } catch (IOException e) {
      throw new RuntimeException(e);
    }
  }

  @Override
  public void copy(String srcKey, String dstKey) {
    Preconditions.checkArgument(!Strings.isNullOrEmpty(srcKey), "Src key should not be empty.");
    File file = path(encode(srcKey)).toFile();

View on GitHub (pinned to 2add963021)

Solutions

  1. Read the message as 'upload directory missing': always call createMultipartUpload(dstKey) first and pass its uploadId together with the exact same dstKey to uploadPartCopy
  2. Log/compare the dstKey string at create and at copy time — they must be byte-identical (same encoding, same prefix, same trailing slash)
  3. After completeUpload or abortMultipartUpload consumed the upload, start a new createMultipartUpload rather than reusing the id
  4. If you wrap ObjectStorage (as PrefixStorage does), ensure the wrapper applies the same prefix transformation on every call

Example fix

// before: upload created for one key, parts copied into another
MultipartUpload u = storage.createMultipartUpload("a/b.txt");
storage.uploadPartCopy("src", "a/c.txt", u.uploadId(), 1, 0, 99); // Upload directory ... already exits (means: missing)
// after: create the upload for the exact destination key
MultipartUpload u = storage.createMultipartUpload(dstKey);
Part p = storage.uploadPartCopy(srcKey, dstKey, u.uploadId(), 1, 0, 99);
Defensive patterns

Strategy: validation

Validate before calling

// create the upload for the exact destination key, then immediately copy parts
MultipartUpload upload = storage.createMultipartUpload(dstKey);
String id = upload.uploadId();
// reuse the SAME dstKey string object/value for every call of this upload
Part p = storage.uploadPartCopy(srcKey, dstKey, id, 1, copyStart, copyEnd);

Try / catch

try {
  Part p = storage.uploadPartCopy(srcKey, dstKey, uploadId, n, start, end);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().contains("already exits")) {
    // message is inverted: it means the upload staging dir is MISSING for this dstKey/uploadId
    throw new IllegalStateException("No multipart upload " + uploadId
        + " for destination key " + dstKey + "; call createMultipartUpload(dstKey) first", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: uploadPartCopy(srcKey, dstKey, uploadId, ...) where no createMultipartUpload(dstKey) with that exact uploadId has run (or it was already completed/aborted), or dstKey differs from the key the upload was created for — including cases where PrefixStorage adds/removes a prefix so the encoded path no longer matches.

Common situations: Copy flows that create the upload for key A but copy parts into key B; using a srcKey-derived uploadId instead of the one from createMultipartUpload(dstKey); prefix-handling wrappers passing keys with inconsistent trailing slashes or encoding; retrying after the upload was completed.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f7d28d9b5e666de9. Report an issue: GitHub.