apache/hadoop · error · IOException

File already exists: %s

Error message

File already exists: %s

What it means

Thrown by SFTPFileSystem.create when the target file already exists on the SFTP server and the overwrite parameter is false. Note the adapter throws a raw IOException rather than FileAlreadyExistsException, so callers that catch the specialized type will miss it.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/sftp/SFTPFileSystem.java:562

   */
  @Override
  public FSDataOutputStream create(Path f, FsPermission permission,
      boolean overwrite, int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    final ChannelSftp client = connect();
    Path workDir;
    try {
      workDir = new Path(client.pwd());
    } catch (SftpException e) {
      throw new IOException(e);
    }
    Path absolute = makeAbsolute(workDir, f);
    if (exists(client, f)) {
      if (overwrite) {
        delete(client, f, false);
      } else {
        disconnect(client);
        throw new IOException(String.format(E_FILE_EXIST, f));
      }
    }
    Path parent = absolute.getParent();
    if (parent == null || !mkdirs(client, parent, FsPermission.getDefault())) {
      parent = (parent == null) ? new Path("/") : parent;
      disconnect(client);
      throw new IOException(String.format(E_CREATE_DIR, parent));
    }
    OutputStream os;
    try {
      final String previousCwd = client.pwd();
      client.cd(parent.toUri().getPath());
      os = client.put(f.getName());
      client.cd(previousCwd);
    } catch (SftpException e) {
      throw new IOException(e);
    }
    FSDataOutputStream fos = new FSDataOutputStream(os, statistics) {

View on GitHub (pinned to 2add963021)

Solutions

1. Pass overwrite=true when the write should replace prior content: fs.create(f, true).
2. Delete the existing file first (fs.delete(f, false)) when you need a deliberate clobber point.
3. If you must detect this case programmatically, catch IOException and test the message for 'File already exists' — the type is not FileAlreadyExistsException in this adapter.

Example fix

// before
FSDataOutputStream out = fs.create(outPath, false, progress); // throws on rerun

// after
FSDataOutputStream out = fs.create(outPath, true, progress);

if (fs.exists(outPath) && !overwrite) {
  throw new IOException("refusing to overwrite existing " + outPath);
}
FSDataOutputStream out = fs.create(outPath, overwrite);

try {
  out = fs.create(outPath, false);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("File already exists")) {
    out = fs.create(outPath, true); // or surface 'output exists' to the user
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling fs.create(f, bufferSize, replication, blockSize, progress) or any overload with overwrite=false when exists(client, f) is true on the sftp:// server; i.e. writing an output path that a previous run already produced.

Common situations: Re-running a failed job without cleaning its output directory; frameworks whose committers rely on FileAlreadyExistsException detection; two writers racing to the same target path.

Related errors

• File already exists: " + f
• {} already exists
• File already exists: ${path}. Append or overwrite option mus
• create(): Mkdirs failed to create: %s
• File exists

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/76aea56353b14d8d. Report an issue: GitHub.