apache/hadoop · error · FileAlreadyExistsException

Cannot create directories because of existing file: %s

Error message

Cannot create directories because of existing file: %s

What it means

GoogleCloudStorageFileSystem.mkdir checks every intermediate directory path of the requested tree via gcs.getItemInfos; if any intermediate path exists as a FILE (a storage object occupying that name), it throws FileAlreadyExistsException naming the offending StorageResourceId. GCS's flat namespace means an object named a/b permanently blocks using a/b as a directory.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageFileSystem.java:695

    // Create a list of all files that can conflict with intermediate/subdirectory paths.
    // For example: gs://foo/bar/zoo/ => (gs://foo/bar, gs://foo/bar/zoo)
    List<StorageResourceId> fileIds =
        getDirs(resourceId.getObjectName()).stream()
            .filter(subdir -> !isNullOrEmpty(subdir))
            .map(
                subdir ->
                    new StorageResourceId(
                        resourceId.getBucketName(), StringPaths.toFilePath(subdir)))
            .collect(toImmutableList());

    // Each intermediate path must ensure that corresponding file does not exist
    //
    // If for any of the intermediate paths file already exists then bail out early.
    // It is possible that the status of intermediate paths can change after
    // we make this check therefore this is a good faith effort and not a guarantee.
    for (GoogleCloudStorageItemInfo fileInfo : gcs.getItemInfos(fileIds)) {
      if (fileInfo.exists()) {
        throw new FileAlreadyExistsException(
            "Cannot create directories because of existing file: " + fileInfo.getResourceId());
      }
    }
  }

  /**
   * For objects whose name looks like a path (foo/bar/zoo), returns all directory paths.
   *
   * <p>For example:
   *
   * <ul>
   *   <li>foo/bar/zoo => returns: (foo/, foo/bar/)
   *   <li>foo/bar/zoo/ => returns: (foo/, foo/bar/, foo/bar/zoo/)
   *   <li>foo => returns: ()
   * </ul>
   *
   * @param objectName Name of an object.
   * @return List of subdirectory like paths.

View on GitHub (pinned to 2add963021)

Solutions

Delete the conflicting file (its StorageResourceId is printed in the exception message) if it is safe to remove.
Rename the conflicting file out of the way, then retry mkdirs.
Choose a different directory name that does not traverse an existing file.
Catch FileAlreadyExistsException and surface which intermediate path collided.

Example fix

// before
fs.mkdirs(new Path("gs://bucket/dir/file.txt/sub"));
// -> FileAlreadyExistsException: existing file: gs://bucket/dir/file.txt

// after
Path conflicting = new Path("gs://bucket/dir/file.txt");
if (fs.exists(conflicting) && !fs.getFileStatus(conflicting).isDirectory()) {
  fs.delete(conflicting, false);
}
fs.mkdirs(new Path("gs://bucket/dir/file.txt/sub"));

Defensive patterns

Strategy: try-catch

Validate before calling

// optional pre-check of ancestors
Path p = new Path("gs://bucket/a/b/c/");
Path anc = p.getParent();
while (anc != null && !anc.isRoot()) {
  if (fs.exists(anc) && !fs.getFileStatus(anc).isDirectory()) {
    throw new IllegalStateException("Ancestor is a file: " + anc);
  }
  anc = anc.getParent();
}

Try / catch

try {
  fs.mkdirs(path);
} catch (FileAlreadyExistsException e) {
  // e.getMessage() names the colliding StorageResourceId; delete/rename that file or pick another directory name
}

Prevention

Keep file names and directory names disjoint in layout schemas (a file at a/b forever blocks directory a/b/...).
Catch FileAlreadyExistsException specifically rather than IOException to surface the colliding resource id.
Clean leftover marker files from earlier runs before mkdirs stages.

When it happens

Trigger: fs.mkdirs(new Path("gs://b/dir/sub/deeper")) when an object named gs://b/dir/sub exists as a file. Any mkdirs whose path goes 'through' an existing file.

Common situations: ETL layouts where the same name is used as a file at one stage and a directory at another; leftover marker/empty files from previous runs occupying directory names; data migrated from HDFS with name collisions.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d014cfcaa73dc48d. Report an issue: GitHub.