apache/hadoop · error · FileAlreadyExistsException

<path> is a directory

Error message

<path> is a directory

What it means

Before writing, create() runs the configured probes (LIST + HEAD via innerGetFileStatus); if the path resolves and the entry is a directory, FileAlreadyExistsException '<path> is a directory' is thrown regardless of the overwrite flag - the source marks it 'automatic error'. S3A never replaces a directory (marker or parent of children) with a file, mirroring POSIX create(2) EISDIR.

Source

Thrown at hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:2168

    }

    // list logic
    boolean skipList = createPerf || magic || cCreate || cEtag;
    if (skipList) {
      probes.remove(StatusProbeEnum.List);
    }

    // if probes are required -request them and evaluate the result.
    if (!probes.isEmpty()) {
      try {

        // get the status or throw an FNFE.
        FileStatus status = innerGetFileStatus(path, false, probes);

        // if the thread reaches here, there is something at the path
        if (status.isDirectory()) {
          // path references a directory: automatic error
          throw new FileAlreadyExistsException(path + " is a directory");
        }
        if (!overwrite) {
          // path references a file and overwrite is disabled
          throw new FileAlreadyExistsException(path + " already exists");
        }
        LOG.debug("Overwriting file {}", path);
      } catch (FileNotFoundException e) {
        // this means there is nothing at the path; all good.
      }
    } else {
      LOG.debug("Skipping all probes with flags:"
              + " createPerf={}, magic={}, ccAvailable={}, cCreate={}, cEtag={}",
          createPerf, magic, ccAvailable, cCreate, cEtag);
    }
    instrumentation.fileCreated();
    final BlockOutputStreamStatistics outputStreamStatistics
        = statisticsContext.newOutputStreamStatistics();
    PutTracker putTracker =

View on GitHub (pinned to 2add963021)

Solutions

1. Choose a different target name for the file so it does not collide with the directory
2. Delete or move the existing directory first: fs.delete(dir, true)
3. Check fs.getFileStatus(path).isDirectory() before create and fail with a clear message

Example fix

// before
FSDataOutputStream out = fs.create(new Path(base, name));

// after
Path target = new Path(base, name);
if (fs.exists(target) && fs.getFileStatus(target).isDirectory()) {
  throw new IllegalArgumentException(target + " is a directory; refusing to replace it");
}
FSDataOutputStream out = fs.create(target, true);

static boolean targetIsDirectory(FileSystem fs, Path p) throws IOException {
  return fs.exists(p) && fs.getFileStatus(p).isDirectory();
}

Catch FileAlreadyExistsException around create() and branch on 'is a directory' in the message: directory collisions are structural (rename/delete or pick a new name), while plain 'already exists' is an overwrite-flag decision.

When it happens

Trigger: fs.create(path) where path is currently a directory, with or without children; overwrite=true does not change the outcome; createPerf/magic paths still hit the probe result when the directory is found.

Common situations: Output file name collides with an existing partition directory (e.g. writing 'data' when 'data/' exists); reruns of a pipeline that previously materialized a directory at the same path; path normalization or trailing-slash differences making a file target resolve to a directory.

Related errors

• <path> already exists
• {} already exists
• {} is a directory
• {} already exists
• File already exists: {file}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d619106c080f2e0. Report an issue: GitHub.