apache/hadoop · error · FileAlreadyExistsException

{} is a directory

Error message

{} is a directory

What it means

In OBSFileSystem.create(...), after getFileStatus succeeds, a directory at the target path is an unconditional error: the code throws FileAlreadyExistsException(f + ' is a directory'). Overwrite=true cannot help — overwriting a directory marker with a file is not a supported object-store operation here. This fires only when something directory-like actually exists at the key; a missing path proceeds to normal creation.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSFileSystem.java:705

      final Path f,
      final FsPermission permission,
      final boolean overwrite,
      final int bufferSize,
      final short replication,
      final long blkSize,
      final Progressable progress)
      throws IOException {
    String key = OBSCommonUtils.pathToKey(this, f);
    FileStatus status;
    long objectLen = 0;
    try {
      // get the status or throw an exception
      status = getFileStatus(f);
      objectLen = status.getLen();
      // if the thread reaches here, there is something at the path
      if (status.isDirectory()) {
        // path references a directory: automatic error
        throw new FileAlreadyExistsException(f + " is a directory");
      }
      if (!overwrite) {
        // path references a file and overwrite is disabled
        throw new FileAlreadyExistsException(f + " already exists");
      }
      LOG.debug("create: Overwriting file {}", f);
    } catch (FileNotFoundException e) {
      // this means the file is not found
      LOG.debug("create: Creating new file {}", f);
    }
    return new FSDataOutputStream(
        new OBSBlockOutputStream(
            this,
            key,
            objectLen,
            new SemaphoredDelegatingExecutor(
                boundedMultipartUploadThreadPool,
                blockOutputActiveBlocks, true),

View on GitHub (pinned to 2add963021)

Solutions

1. Target the file path, not its parent directory (e.g. /out/part-r-00000, not /out)
2. Before create, if overwrite is on and the path is a directory, delete it: if (fs.exists(p) && fs.getFileStatus(p).isDirectory()) fs.delete(p, true)
3. Validate the target with getFileStatus and fail with a clear application error naming the conflicting directory
4. Clean stale output directories in job setup rather than reusing them as file targets

Example fix

// before
fs.create(new Path("/job/out"), true); // /job/out is an existing directory -> FileAlreadyExistsException

// after
Path out = new Path("/job/out");
if (fs.exists(out) && fs.getFileStatus(out).isDirectory()) {
  fs.delete(out, true);
}
FSDataOutputStream os = fs.create(out, true);

if (fs.exists(f)) {
  FileStatus st = fs.getFileStatus(f);
  if (st.isDirectory()) {
    if (overwrite) {
      fs.delete(f, true); // only if semantically safe
    } else {
      throw new FileAlreadyExistsException("target is a directory: " + f);
    }
  }
}
FSDataOutputStream out = fs.create(f, overwrite);

try {
  return fs.create(f, overwrite);
} catch (FileAlreadyExistsException e) {
  if (String.valueOf(e.getMessage()).endsWith("is a directory")) {
    // overwrite cannot fix this: delete the directory or pick a file path
  }
  throw e;
}

When it happens

Trigger: Calling fs.create(path, ...) where path was previously created by mkdirs or by writing another object under it (implied directory); output paths colliding with existing directory structures (e.g. re-running a job whose outputDir itself is passed instead of outputDir/part-0); CreateFlag.OVERWRITE set but the existing entry is a directory.

Common situations: Specifying the MapReduce/Spark output directory itself as the output file; leftover directory markers from prior runs blocking new file creation; path normalization bugs that drop the filename and leave the parent directory as the target.

Related errors

• Can't open {} because it is a directory
• Not a directory: {}
• {} already exists
• {} already exists
• Cannot create a file whose name looks like a directory: '%s'

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8339480384288a08. Report an issue: GitHub.