What causes "Failed to create MultipartUpload with key: %s" in apache/hadoop?

Inspect and repair the staging tree: ls -lR /__STAGING__ and remove stray regular files blocking directory creation

apache/hadoop · error · RuntimeException

Failed to create MultipartUpload with key: %s

Error message

Failed to create MultipartUpload with key: %s

What it means

FileStore.createMultipartUpload() represents an upload as a staging directory <root>/__STAGING__/<encodedKey>/<uploadId>. If uploadDir.mkdirs() returns false the RuntimeException 'Failed to create MultipartUpload with key' is thrown — the connector could not create the per-upload staging directory on local disk, so the multipart upload cannot even begin.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java:335

          }

          objectInfos.add(obj);
        }
      }
      return new ListObjectsResponse(objectInfos, new ArrayList<>(commonPrefixes));
    }
  }

  @Override
  public MultipartUpload createMultipartUpload(String key) {
    Preconditions.checkArgument(!Strings.isNullOrEmpty(key), "Key should not be empty.");

    String uploadId = UUIDUtils.random();
    Path uploadDir = uploadPath(key, uploadId);
    if (uploadDir.toFile().mkdirs()) {
      return new MultipartUpload(key, uploadId, MIN_PART_SIZE, MAX_PART_COUNT);
    } else {
      throw new RuntimeException("Failed to create MultipartUpload with key: " + key);
    }
  }

  private Path uploadPath(String key, String uploadId) {
    return Paths.get(root, STAGING_DIR, encode(key), uploadId);
  }

  @Override
  public Part uploadPart(
      String key, String uploadId, int partNum,
      InputStreamProvider streamProvider, long contentLength) {
    Preconditions.checkArgument(!Strings.isNullOrEmpty(key), "Key should not be empty.");

    File uploadDir = uploadPath(key, uploadId).toFile();
    if (!uploadDir.exists()) {
      throw new RuntimeException("cannot locate the upload id: " + uploadId);
    }

View on GitHub (pinned to 2add963021)

Solutions

Inspect and repair the staging tree: ls -lR <root>/__STAGING__ and remove stray regular files blocking directory creation
Verify the root volume is writable and has space (df -h <root>; touch <root>/.probe)
Recreate the store root directory if it was deleted post-initialize, or re-initialize the FileStore
Delete the __STAGING__ subtree entirely (it only holds in-flight uploads) and retry the upload

Defensive patterns

Strategy: validation

Validate before calling

// pre-create the staging tree so createMultipartUpload only makes leaf dirs
java.nio.file.Path staging = java.nio.file.Paths.get(root, "__STAGING__");
java.nio.file.Files.createDirectories(staging);
if (!java.nio.file.Files.isWritable(staging)) {
  throw new IOException("filestore staging dir not writable: " + staging);
}

Prevention

Keep the store root on a writable local volume and never delete it while the store is active
Wipe <root>/__STAGING__ between test runs only when no upload is in flight
Avoid creating plain files that collide with __STAGING__ paths (the encoded-key path must be a directory)
Re-initialize the FileStore after anything deletes the root directory

When it happens

Trigger: createMultipartUpload(key) when any path component under root/__STAGING__/<encoded key> cannot be created: root or staging not writable, a regular file sitting where a directory is needed, disk full, or the store root itself was deleted after initialize().

Common situations: A previous aborted job left a FILE (not directory) at the encoded-key path inside __STAGING__; read-only or full volumes in test containers; cleanup code that wipes the root between operations while the filesystem layer still holds a FileStore instance.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5222a646b90eb466. Report an issue: GitHub.