apache/hadoop · error · FileAlreadyExistsException

{} already exists

Error message

{} already exists

What it means

CosNFileSystem.create throws FileAlreadyExistsException when the target path already exists as a file in the COS bucket and the caller did not set overwrite. Before creating the object, create() calls getFileStatus; a directory at the path produces the sibling 'is a directory' error, while an existing file with overwrite=false produces this one. This follows the Hadoop FileSystem contract that existing data must never be silently clobbered.

Source

Thrown at hadoop-cloud-storage-project/hadoop-cos/src/main/java/org/apache/hadoop/fs/cosn/CosNFileSystem.java:252

  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress) throws IOException {
    throw new IOException("Not supported");
  }

  @Override
  public FSDataOutputStream create(Path f, FsPermission permission,
      boolean overwrite, int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    FileStatus fileStatus;

    try {
      fileStatus = getFileStatus(f);
      if (fileStatus.isDirectory()) {
        throw new FileAlreadyExistsException(f + " is a directory");
      }
      if (!overwrite) {
        // path references a file and overwrite is disabled
        throw new FileAlreadyExistsException(f + " already exists");
      }

    } catch (FileNotFoundException e) {
      LOG.debug("Creating a new file: [{}] in COS.", f);
    }

    Path absolutePath = makeAbsolute(f);
    String key = pathToKey(absolutePath);
    return new FSDataOutputStream(
        new CosNOutputStream(getConf(), store, key, blockSize,
            this.boundedIOThreadPool), statistics);
  }

  private boolean rejectRootDirectoryDelete(boolean isEmptyDir,
      boolean recursive) throws PathIOException {
    if (isEmptyDir) {
      return true;
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Pass overwrite=true: fs.create(path, true), or include CreateFlag.OVERWRITE for createNonRecursive.
2. Delete the stale target first: if (fs.exists(p) && fs.getFileStatus(p).isFile()) fs.delete(p, false);
3. Write to a unique output path per run (jobId/timestamp suffix) or let the output committer promote files only on commit.
4. If the existing file is unexpected, inspect getFileStatus(p).getModificationTime() to find which process created it before deleting.

Example fix

// before
FSDataOutputStream out = fs.create(outPath, false); // FileAlreadyExistsException: ... already exists

// after
if (fs.exists(outPath) && fs.getFileStatus(outPath).isFile()) {
  fs.delete(outPath, false);
}
FSDataOutputStream out = fs.create(outPath, true);

Path p = new Path('cosn://bucket/out/part-00000');
boolean overwriteRequested = true;
if (fs.exists(p) && fs.getFileStatus(p).isFile() && !overwriteRequested) {
  throw new IllegalStateException('Refusing to overwrite existing file: ' + p);
}

static boolean isAlreadyExists(Throwable t) {
  return t instanceof FileAlreadyExistsException;
}

try {
  try (FSDataOutputStream out = fs.create(p, false)) { /* write */ }
} catch (FileAlreadyExistsException e) {
  // expected on rerun: clean up and retry once, or treat as no-op
  if (allowOverwrite) { fs.delete(p, false); /* retry create once */ } else { throw e; }
}

When it happens

Trigger: fs.create(path, false); fs.create(path, permission, false /*overwrite*/, ...) ; createNonRecursive with an EnumSet that lacks CreateFlag.OVERWRITE; any committer or writer (FileOutputCommitter, Spark _temporary writes, Hive staged writes) targeting a path that already holds a file.

Common situations: Re-running a MapReduce/Spark/Hive job into the same output directory without cleaning it; a previous run crashed after committing output; checkpoint files already present; passing the overwrite argument in the wrong positional slot so it lands as false.

Related errors

• Not a directory: {}
• Path is a file: {}
• Can't make directory for path '%s' since it is a file.
• File: %s already exists
• {} is a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8501e49921872f60. Report an issue: GitHub.