apache/hadoop · error · RuntimeException

cannot locate the upload id: %s

Error message

cannot locate the upload id: %s

What it means

FileStore.uploadPart() resolves the upload's staging directory <root>/__STAGING__/<encodedKey>/<uploadId> and throws 'cannot locate the upload id' when that directory does not exist. In this local emulation the uploadId is only valid as long as its staging directory exists — once completeUpload or abortMultipartUpload deletes it (or the root changes), the id is dead.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java:351

      return new MultipartUpload(key, uploadId, MIN_PART_SIZE, MAX_PART_COUNT);
    } else {
      throw new RuntimeException("Failed to create MultipartUpload with key: " + key);
    }
  }

  private Path uploadPath(String key, String uploadId) {
    return Paths.get(root, STAGING_DIR, encode(key), uploadId);
  }

  @Override
  public Part uploadPart(
      String key, String uploadId, int partNum,
      InputStreamProvider streamProvider, long contentLength) {
    Preconditions.checkArgument(!Strings.isNullOrEmpty(key), "Key should not be empty.");

    File uploadDir = uploadPath(key, uploadId).toFile();
    if (!uploadDir.exists()) {
      throw new RuntimeException("cannot locate the upload id: " + uploadId);
    }

    File partFile = new File(uploadDir, String.valueOf(partNum));
    copyInputStreamToFile(streamProvider.newStream(), partFile, contentLength);

    try {
      byte[] data = Files.readAllBytes(partFile.toPath());
      return new Part(partNum, data.length, DigestUtils.md5Hex(data));
    } catch (IOException e) {
      LOG.error("failed to locate the part file: {}", partFile.getAbsolutePath());
      throw new RuntimeException(e);
    }
  }

  private static void appendInputStreamToFile(InputStream in, File partFile, long contentLength) {
    try (FileOutputStream out = new FileOutputStream(partFile, true)) {
      long copiedBytes = IOUtils.copyLarge(in, out, 0, contentLength);

View on GitHub (pinned to 2add963021)

Solutions

  1. Only use uploadId values returned by createMultipartUpload() on the same ObjectStorage instance/root, within that upload's lifetime
  2. After a completeUpload or abortMultipartUpload, always start a fresh createMultipartUpload instead of reusing the old id
  3. Make sure every process/test resolving the store uses the same root configuration
  4. Guard async part writers against cleanup races: shut them down before deleting the staging/root directories

Example fix

// before: stale id from an earlier (already completed) upload
storage.uploadPart(key, oldUploadId, 1, provider, len); // RuntimeException: cannot locate the upload id
// after: create and use the upload in one lifecycle
MultipartUpload upload = storage.createMultipartUpload(key);
Part part = storage.uploadPart(key, upload.uploadId(), 1, provider, len);
storage.completeUpload(key, upload.uploadId(), List.of(part));
Defensive patterns

Strategy: validation

Validate before calling

// track the upload lifecycle in the calling code
java.util.Set<String> activeUploads = java.util.concurrent.ConcurrentHashMap.newKeySet();
MultipartUpload upload = storage.createMultipartUpload(key);
activeUploads.add(upload.uploadId());

// later, before uploading a part:
if (!activeUploads.contains(uploadId)) {
  throw new IllegalStateException("uploadId not active (completed/aborted?): " + uploadId);
}
storage.uploadPart(key, uploadId, partNum, provider, len);

Prevention

When it happens

Trigger: uploadPart(key, uploadId, partNum, ...) with an uploadId that was never created by createMultipartUpload in this store, one whose upload was already completed or aborted (staging dir removed), or when the process resolves a different fs.filestore.endpoint/FILE_STORAGE_ROOT than the one that created the upload.

Common situations: Retrying a part upload after the upload was already completed or aborted; tests that create the FileStore per test method but share uploadIds across tests; cleanup (@After) deleting the staging tree while async part writers are still running; hand-copied uploadId strings.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fb2a821bfe654598. Report an issue: GitHub.