apache/hadoop · error · IOException

Fail to rename tmp file (={tmp}) to destination file (={dst}

Error message

Fail to rename tmp file (={tmp}) to destination file (={dst})

What it means

CopyCommitter.concatFileChunks() (used with the -blocksperchunk chunked-copy mode) concatenates copied chunks and renames the result onto the target file through this rename() helper. If exists(), delete() or rename() on the destination throws, the IOException is re-wrapped with the tmp and destination paths. The real failure is in the nested cause.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:674

            targetFile, skipCrc, srcFileStatus.getLen());
  }

  /**
   * Rename tmp to dst on destFileSys.
   * @param destFileSys the file ssystem
   * @param tmp the source path
   * @param dst the destination path
   * @throws IOException if renaming failed
   */
  private static void rename(FileSystem destFileSys, Path tmp, Path dst)
      throws IOException {
    try {
      if (destFileSys.exists(dst)) {
        destFileSys.delete(dst, true);
      }
      destFileSys.rename(tmp, dst);
    } catch (IOException ioe) {
      throw new IOException("Fail to rename tmp file (=" + tmp
          + ") to destination file (=" + dst + ")", ioe);
    }
  }

}

View on GitHub (pinned to 2add963021)

Solutions

1. Walk the nested cause in the job log to see whether delete or rename failed and why
2. Ensure only one job writes a given target file at a time; serialize distcp runs that share a destination
3. Stop or scope cleanup jobs so they cannot delete .distcp.tmp.* or chunk files while distcp runs
4. Re-run with -update so completed files are skipped and only the failed target is re-copied

catch (IOException e) {  // wraps 'Fail to rename tmp file'
  Throwable root = e;
  while (root.getCause() != null) root = root.getCause();
  LOG.error("chunk concat/rename failed, root cause:", root);
}

When it happens

Trigger: Another writer holds or recreates the destination between the delete and the rename; permission lost on the destination parent; the tmp chunk file removed by an external cleaner; two concurrent distcp jobs committing the same target file.

Common situations: Overlapping distcp runs with -blocksperchunk aimed at the same destination; cron cleanup deleting .distcp.tmp.* files mid-commit; ACL or ownership changes on the target directory during a long job.

Related errors

• blocksperchunk is not supported since the target file system
• Failed to concat chunk files for {targetFile}
• Inconsistent sequence file: current chunk file {srcFileStatu
• Target-path can't be committed to because it exists at {fina
• Atomic commit failed. Temporary data in {workDir}, Unable to

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/479a752a7b3d8b14. Report an issue: GitHub.