What causes "File already exists: {file}" in apache/hadoop?

Pass overwrite=true when re-writing is intended: fs.create(path, true)

apache/hadoop · error · FileAlreadyExistsException

File already exists: {file}

Error message

File already exists: {file}

What it means

create() stats the target first. If something exists there and either overwrite is false or the existing entry is a directory (overwrite only deletes files), FileAlreadyExistsException is thrown after disconnecting the client. Directories are never overwritten by create().

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java:330

  @Override
  public FSDataOutputStream create(Path file, FsPermission permission,
      boolean overwrite, int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    final FTPClient client = connect();
    Path workDir = new Path(client.printWorkingDirectory());
    Path absolute = makeAbsolute(workDir, file);
    FileStatus status;
    try {
      status = getFileStatus(client, file);
    } catch (FileNotFoundException fnfe) {
      status = null;
    }
    if (status != null) {
      if (overwrite && !status.isDirectory()) {
        delete(client, file, false);
      } else {
        disconnect(client);
        throw new FileAlreadyExistsException("File already exists: " + file);
      }
    }
    
    Path parent = absolute.getParent();
    if (parent == null || !mkdirs(client, parent, FsPermission.getDirDefault())) {
      parent = (parent == null) ? new Path("/") : parent;
      disconnect(client);
      throw new IOException("create(): Mkdirs failed to create: " + parent);
    }
    client.allocate(bufferSize);
    // Change to parent directory on the server. Only then can we write to the
    // file on the server by opening up an OutputStream. As a side effect the
    // working directory on the server is changed to the parent directory of the
    // file. The FTP client connection is closed when close() is called on the
    // FSDataOutputStream.
    client.changeWorkingDirectory(parent.toUri().getPath());
    OutputStream outputStream = client.storeFileStream(file.getName());

View on GitHub (pinned to 2add963021)

Solutions

Pass overwrite=true when re-writing is intended: fs.create(path, true)
If the target is a directory, delete it first: fs.delete(path, true)
Catch FileAlreadyExistsException to implement idempotent create-replace (delete then retry once)
Clean output paths in job setup instead of relying on create failing

Example fix

// before
FSDataOutputStream out = fs.create(path, false);
// FileAlreadyExistsException

// after
try {
  out = fs.create(path, true);
} catch (FileAlreadyExistsException e) {
  fs.delete(path, false);
  out = fs.create(path, true);
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (fs.exists(path) && !overwrite) {
  // decide before create(): fail, skip, or delete
  fs.delete(path, false);
}
FSDataOutputStream out = fs.create(path, true);

Try / catch

try {
  out = fs.create(path, true);
} catch (FileAlreadyExistsException e) {
  // idempotent create: clear the conflicting entry (file or dir) and retry once
  fs.delete(path, true);
  out = fs.create(path, true);
}

Prevention

Pass overwrite=true explicitly whenever re-running jobs is expected
Clean output paths in job initialization instead of relying on create() to fail
Remember overwrite never replaces an existing directory — delete it explicitly

When it happens

Trigger: fs.create(path) or fs.create(path, false) on an existing file; fs.create(path, true) where the path is an existing directory; two writers racing to create the same output path.

Common situations: Job re-runs without cleaning output; part files left by a failed attempt; the overwrite flag dropped when wrapping create calls; a directory accidentally created at the intended file path.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/23e20bc09239898c. Report an issue: GitHub.