What causes "File {f} already exists" in apache/hadoop?

Delete or archive the existing file, then retry the call

apache/hadoop · error · AlreadyExistsException

File already exists

Error message

File {f} already exists

What it means

When secure O_EXCL creation is unavailable (security disabled or native code missing, i.e. skipSecurity==true), SecureIOUtils.createForWrite falls back to insecureCreateForWrite: a racy exists() check followed by open+chmod. If the path exists at check time it throws AlreadyExistsException (an IOException subclass defined inside SecureIOUtils). The check is best-effort only; a file created between check and open would still be clobbered, which is why this path exists solely for the insecure mode.

Solutions

Delete or archive the existing file, then retry the call
Write to a unique name per attempt (append attempt id or UUID) instead of a fixed path
Clean the node's local scratch dirs (nm-local-dirs, mapreduce local dirs) when re-running jobs
Run with security enabled plus native libraries so the secure O_EXCL path is used instead of this racy fallback

Example fix

// before
FileOutputStream fos = SecureIOUtils.createForWrite(f, 0644); // AlreadyExistsException on stale file

// after
try {
  fos = SecureIOUtils.createForWrite(f, 0644);
} catch (SecureIOUtils.AlreadyExistsException e) {
  throw new IOException("Stale output " + f + " already present; clean local dirs", e);
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (f.exists()) {
  // stale output: clean up or pick a unique name before calling createForWrite
  throw new IOException(f + " already exists; remove it or use a unique attempt path");
}

Try / catch

try {
  FileOutputStream fos = SecureIOUtils.createForWrite(f, perms);
} catch (SecureIOUtils.AlreadyExistsException e) {
  // recover: delete stale file, or retry with a unique name
  f.delete();
  fos = SecureIOUtils.createForWrite(f, perms);
}

Prevention

Always derive output filenames from attempt/task ids so reruns cannot collide
Clean local scratch dirs between job reruns
Remember the insecure path is racy: under disabled security, catching this exception is a best-effort guard, not an atomic create

When it happens

Trigger: SecureIOUtils.createForWrite(path, permissions) with security off/native libs absent while path already exists: leftover files from a previous task attempt in local dirs, two processes racing for the same filename, or re-running a job against uncleaned scratch space.

Common situations: Retried map/reduce tasks reusing mapreduce.cluster.local.dir; NodeManager local dirs not cleaned after an unclean kill; duplicated attempt ids; manual test runs that left files behind.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20f5ea6f36e00712. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java:248

    try {
      Stat stat = NativeIO.POSIX.getFstat(fis.getFD());
      checkStat(f, stat.getOwner(), stat.getGroup(), expectedOwner,
          expectedGroup);
      success = true;
      return fis;
    } finally {
      if (!success) {
        fis.close();
      }
    }
  }

  private static FileOutputStream insecureCreateForWrite(File f,
      int permissions) throws IOException {
    // If we can't do real security, do a racy exists check followed by an
    // open and chmod
    if (f.exists()) {
      throw new AlreadyExistsException("File " + f + " already exists");
    }
    FileOutputStream fos = new FileOutputStream(f);
    boolean success = false;
    try {
      rawFilesystem.setPermission(new Path(f.getAbsolutePath()),
        new FsPermission((short)permissions));
      success = true;
      return fos;
    } finally {
      if (!success) {
        fos.close();
      }
    }
  }

  /**
   * Open the specified File for write access, ensuring that it does not exist.
   * @param f the file that we want to create

View on GitHub (pinned to 2add963021)