apache/hadoop · error · IOException

rename from {} to {} failed.

Error message

rename from {} to {} failed.

What it means

IOException thrown at the end of a failed two-phase rename after the NameNode already rolled back (tx.restoreSource/restoreDst in the finally block). It means the source was detached but re-attaching it at the destination failed - classically another client created dst concurrently in the non-overwrite path - so the namespace was restored to the pre-rename state and the failure reported. State stays consistent; the operation just lost a race.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java:497

          // deleted. Need to update the SnapshotManager.
          fsd.getFSNamesystem().removeSnapshottableDirs(dstSnapshottableDirs);
        }

        tx.updateQuotasInSourceTree(bsps);
        return createRenameResult(
            fsd, renamedIIP, filesDeleted, collectedBlocks);
      }
    } finally {
      if (undoRemoveSrc) {
        tx.restoreSource();
      }
      if (undoRemoveDst) { // Rename failed - restore dst
        tx.restoreDst(bsps);
      }
    }
    NameNode.stateChangeLog.warn("DIR* FSDirectory.unprotectedRenameTo: " +
        "failed to rename " + src + " to " + dst);
    throw new IOException("rename from " + src + " to " + dst + " failed.");
  }

  /**
   * @deprecated Use {@link #renameToInt(FSDirectory, FSPermissionChecker,
   * String, String, boolean, Options.Rename...)}
   */
  @Deprecated
  private static RenameResult renameTo(FSDirectory fsd, FSPermissionChecker pc,
      INodesInPath srcIIP, INodesInPath dstIIP, boolean logRetryCache)
          throws IOException {
    if(fsd.isNonEmptyDirectory(srcIIP)) {
      DFSUtil.checkProtectedDescendants(fsd, srcIIP);
    }

    if (fsd.isPermissionEnabled()) {
      // Check write access to parent of src
      fsd.checkPermission(pc, srcIIP, false, null, FsAction.WRITE, null, null,
          false);

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the rename after re-resolving; the race has usually cleared.
2. If the destination legitimately exists, use the overwrite variant: fs.rename(src, dst, Options.Rename.OVERWRITE).
3. Design commit paths so only one writer performs the final rename (unique temp names plus a single renamer).

Example fix

// before
fs.rename(src, dst); // IOException: rename from X to Y failed (lost race)

// after
for (int i = 0; i < 3; i++) {
  try { fs.rename(src, dst); break; }
  catch (IOException e) {
    if (!fs.exists(src) || i == 2) throw e;
    if (fs.exists(dst)) { fs.rename(src, dst, Options.Rename.OVERWRITE); break; }
  }
}

static void renameWithRetry(FileSystem fs, Path src, Path dst, int attempts) throws IOException {
  for (int i = 0; i < attempts; i++) {
    try {
      fs.rename(src, dst);
      return;
    } catch (IOException e) {
      if (!fs.exists(src)) throw e;          // source really gone: not a race
      if (fs.exists(dst)) {                  // destination appeared concurrently
        fs.rename(src, dst, Options.Rename.OVERWRITE);
        return;
      }
      if (i == attempts - 1) throw e;
    }
  }
}

When it happens

Trigger: Two clients renaming to the same destination concurrently; commit protocols where multiple tasks race the final rename to a common path; overwrite=false while the destination appeared between validation and addLastINode.

Common situations: Atomic-output commit races (two tasks both attempt the final move); distcp or balancer-like tooling contending with live jobs; aggressive client retries that recreate the destination.

Related errors

• {} doesn't support renameSnapshot
• Could not rename {oldDir} to {newDir}
• Cannot delete/rename non-empty protected directory {}
• Cannot delete/rename non-empty protected subdirectory {}
• Cannot delete/rename subdirectory under protected subdirecto

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9920f1269c01126e. Report an issue: GitHub.