apache/hadoop · error · FileAlreadyExistsException

<src> for client <clientMachine> already exists

Error message

<src> for client <clientMachine> already exists

What it means

startFile without OVERWRITE on a file that is open for write goes through lease handling: recoverLeaseInternal throws AlreadyBeingCreatedException when another client holds the lease within the soft limit, and when the soft limit has passed it recovers the lease and then throws FileAlreadyExistsException so the client can retry. Either way the path exists and is under construction by a live or recently-live writer.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java:396

    if (iip.getLastINode() != null) {
      fsn.writeLock(RwLockMode.BM);
      try {
        if (overwrite) {
          List<INode> toRemoveINodes = new ChunkedArrayList<>();
          List<Long> toRemoveUCFiles = new ChunkedArrayList<>();
          long ret = FSDirDeleteOp.delete(fsd, iip, toRemoveBlocks,
              toRemoveINodes, toRemoveUCFiles, now());
          if (ret >= 0) {
            iip = INodesInPath.replace(iip, iip.length() - 1, null);
            FSDirDeleteOp.incrDeletedFileCount(ret);
            fsn.removeLeasesAndINodes(toRemoveUCFiles, toRemoveINodes, true);
          }
        } else {
          // If lease soft limit time is expired, recover the lease
          fsn.recoverLeaseInternal(FSNamesystem.RecoverLeaseOp.CREATE_FILE, iip,
              src, holder, clientMachine, false);
          throw new FileAlreadyExistsException(src + " for client " +
              clientMachine + " already exists");
        }
      } finally {
        fsn.writeUnlock(RwLockMode.BM, "create");
      }
    }
    fsn.checkFsObjectLimit();
    INodeFile newNode = null;
    INodesInPath parent =
        FSDirMkdirOp.createAncestorDirectories(fsd, iip, permissions);
    if (parent != null) {
      iip = addFile(fsd, parent, iip.getLastLocalName(), permissions,
          replication, blockSize, holder, clientMachine, shouldReplicate,
          ecPolicyName, storagePolicy);
      newNode = iip != null ? iip.getLastINode().asFile() : null;
    }
    if (newNode == null) {
      throw new IOException("Unable to add " + src +  " to namespace");

View on GitHub (pinned to 2add963021)

Solutions

1. Wait for the lease to clear (soft limit 60 s, hard limit 1 h) or force it with 'hdfs debug recoverLease -path <src>' or DistributedFileSystem.recoverLease(), then retry
2. Give each writer attempt a unique file name so writers never collide
3. If replacing the file is intended, create with overwrite=true

Example fix

// before
FSDataOutputStream out = fs.create(path, false); // throws while old lease lives

// after: recover a stale lease from a dead writer, then create
DistributedFileSystem dfs = (DistributedFileSystem) fs;
while (!dfs.isFileClosed(path)) {
  dfs.recoverLease(path);
  Thread.sleep(1_000L);
}
FSDataOutputStream out = dfs.create(path, false);

DistributedFileSystem dfs = (DistributedFileSystem) fs;
if (!dfs.isFileClosed(path)) {
  dfs.recoverLease(path);
  // poll until closed, bounded by lease soft/hard limits
}

try {
  out = fs.create(path, false);
} catch (FileAlreadyExistsException | AlreadyBeingCreatedException e) {
  Thread.sleep(5_000L);
  out = fs.create(path, false); // retry after the lease clears
}

When it happens

Trigger: fs.create(path) or fs.append(path) without overwrite while the same file is open for write by another client, or by a crashed client whose lease is younger than dfs.namenode.lease-soft-limit-sec (default 60 s).

Common situations: Application restart reopening the same output file within a minute of the previous instance dying; speculative or duplicated task attempts writing the same HDFS path; two services configured with the same output file.

Related errors

• Failed to {} {} for {} on {} because {} is already the curre
• Failed to {} {} for {} on {} because this file lease is curr
• Client (={}) is not the lease owner (={}: {} (inode {}) {}
• A directory with that name exists: %s
• {} already exists

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b89f5011489b62a2. Report an issue: GitHub.