apache/hadoop · error · AlreadyExistsException

File {f} already exists

Error message

File {f} already exists

What it means

When secure O_EXCL creation is unavailable (security disabled or native code missing, i.e. skipSecurity==true), SecureIOUtils.createForWrite falls back to insecureCreateForWrite: a racy exists() check followed by open+chmod. If the path exists at check time it throws AlreadyExistsException (an IOException subclass defined inside SecureIOUtils). The check is best-effort only; a file created between check and open would still be clobbered, which is why this path exists solely for the insecure mode.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java:248

    try {
      Stat stat = NativeIO.POSIX.getFstat(fis.getFD());
      checkStat(f, stat.getOwner(), stat.getGroup(), expectedOwner,
          expectedGroup);
      success = true;
      return fis;
    } finally {
      if (!success) {
        fis.close();
      }
    }
  }

  private static FileOutputStream insecureCreateForWrite(File f,
      int permissions) throws IOException {
    // If we can't do real security, do a racy exists check followed by an
    // open and chmod
    if (f.exists()) {
      throw new AlreadyExistsException("File " + f + " already exists");
    }
    FileOutputStream fos = new FileOutputStream(f);
    boolean success = false;
    try {
      rawFilesystem.setPermission(new Path(f.getAbsolutePath()),
        new FsPermission((short)permissions));
      success = true;
      return fos;
    } finally {
      if (!success) {
        fos.close();
      }
    }
  }

  /**
   * Open the specified File for write access, ensuring that it does not exist.
   * @param f the file that we want to create

View on GitHub (pinned to 2add963021)

Solutions

1. Delete or archive the existing file, then retry the call
2. Write to a unique name per attempt (append attempt id or UUID) instead of a fixed path
3. Clean the node's local scratch dirs (nm-local-dirs, mapreduce local dirs) when re-running jobs
4. Run with security enabled plus native libraries so the secure O_EXCL path is used instead of this racy fallback

Example fix

// before
FileOutputStream fos = SecureIOUtils.createForWrite(f, 0644); // AlreadyExistsException on stale file

// after
try {
  fos = SecureIOUtils.createForWrite(f, 0644);
} catch (SecureIOUtils.AlreadyExistsException e) {
  throw new IOException("Stale output " + f + " already present; clean local dirs", e);
}

if (f.exists()) {
  // stale output: clean up or pick a unique name before calling createForWrite
  throw new IOException(f + " already exists; remove it or use a unique attempt path");
}

try {
  FileOutputStream fos = SecureIOUtils.createForWrite(f, perms);
} catch (SecureIOUtils.AlreadyExistsException e) {
  // recover: delete stale file, or retry with a unique name
  f.delete();
  fos = SecureIOUtils.createForWrite(f, perms);
}

When it happens

Trigger: SecureIOUtils.createForWrite(path, permissions) with security off/native libs absent while path already exists: leftover files from a previous task attempt in local dirs, two processes racing for the same filename, or re-running a job against uncleaned scratch space.

Common situations: Retried map/reduce tasks reusing mapreduce.cluster.local.dir; NodeManager local dirs not cleaned after an unclean kill; duplicated attempt ids; manual test runs that left files behind.

Related errors

• Multipart upload incomplete: expected {} parts but got {}
• Can't make directory for path: %s, since it is a file.
• No URI in deserialized Path
• hadoop.security.authorizationis configured to true but servi
• Secure IO is not possible without native code extensions.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20f5ea6f36e00712. Report an issue: GitHub.