apache/hadoop · error · IOException

Failed to rename {} to {}

Error message

Failed to rename {} to {}

What it means

In the same trash-restore loop, renaming a trashed block file back to its restore location failed (File.renameTo returned false) — the block file could not be moved back into current/, so the rollback aborts.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java:582

      }

      if (restoreDirectory == null) {
        restoreDirectory = new File(getRestoreDirectory(child));
        if (!restoreDirectory.exists() && !restoreDirectory.mkdirs()) {
          throw new IOException("Failed to create directory " + restoreDirectory);
        }
      }

      final File newChild = new File(restoreDirectory, child.getName());

      if (newChild.exists() && newChild.length() >= child.length()) {
        // Failsafe - we should not hit this case but let's make sure
        // we never overwrite a newer version of a block file with an
        // older version.
        LOG.info("Not overwriting {} with smaller file from " +
            "trash directory. This message can be safely ignored.", newChild);
      } else if (!child.renameTo(newChild)) {
        throw new IOException("Failed to rename " + child + " to " + newChild);
      } else {
        ++filesRestored;
      }
    }
    FileUtil.fullyDelete(trashRoot);
    return filesRestored;
  }

  /*
   * Roll back to old snapshot at the block pool level
   * If previous directory exists: 
   * <ol>
   * <li>Rename <SD>/current/<bpid>/current to removed.tmp</li>
   * <li>Rename * <SD>/current/<bpid>/previous to current</li>
   * <li>Remove removed.tmp</li>
   * </ol>
   * 
   * Do nothing if previous directory does not exist.

View on GitHub (pinned to 2add963021)

Solutions

1. Verify the DN user owns both the trash entry and the target dir and nothing holds the file open (lsof)
2. Copy the file manually (cp, then confirm identical length) and delete the trash copy if rename keeps failing
3. Fix the underlying FS issue (space, mount flags) and retry the rollback

Example fix

# cp is a safe manual fallback to rename during trash restore:
cp -p /dfs/dn/current/BP-*/trash/subdir/blk_x /dfs/dn/current/BP-*/current/finalized/subdir/blk_x
cmp blk_x blk_x   # lengths/bytes equal
hdfs --daemon start datanode -rollback

catch (IOException e) {
  if (e.getMessage().contains("Failed to rename")) {
    // fall back to copy+verify+delete, which achieves the same restore
    restoreByCopy(child, newChild);
  }
}

When it happens

Trigger: Permission denied on source or target; another process holds the file open; target anomaly or filesystem (NFS/FUSE) refusing the rename; out-of-space on metadata updates.

Common situations: Rolling-upgrade rollback with AV/scanners or backup agents touching block files; DN user mismatch; NAS-backed storage.

Related errors

• Cannot remove directory {}
• Cannot remove directory {}
• Failed to create directory {}
• Storage directory with location {} does not exist
• Detached directory {} is not empty. Please manually move eac

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/756b1169e7368e01. Report an issue: GitHub.