apache/hadoop · error · IOException

Target-path can't be committed to because it exists at

Error message

Target-path can't be committed to because it exists at {finalDir}. Copied data is in temp-dir: {workDir}. 

What it means

Thrown by DistCp's CopyCommitter during the final atomic-commit step (hadoop distcp -atomic). commitData() reads the temp dir from distcp.target.work.path and the destination from distcp.target.final.path; before renaming work onto final it checks targetFS.exists(finalDir), and when the final path already exists alongside the work dir it aborts the commit so pre-existing data is never silently overwritten. The freshly copied data stays safe in the temp work directory named in the message.

Solutions

1. Move or delete the existing final directory (hdfs dfs -mv <finalDir> <finalDir>.bak) and re-run the distcp job
2. If the existing finalDir holds stale data, delete it and recover the new copy from the temp work dir printed in the error (hdfs dfs -mv <workDir> <finalDir>)
3. Verify nothing else creates the target while the job runs; aim -atomic runs at a fresh destination path
4. If replacing an existing target is the goal, drop -atomic and use -overwrite instead

Example fix

# before: /data/final already exists, so commitData() aborts
hadoop distcp -atomic hdfs://nn/src hdfs://nn/data/final

# after: move the old target aside, then re-run
hdfs dfs -mv hdfs://nn/data/final hdfs://nn/data/final.bak
hadoop distcp -atomic hdfs://nn/src hdfs://nn/data/final

// before launching an atomic distcp, assert the final path is free
FileSystem tfs = finalDir.getFileSystem(conf);
if (tfs.exists(finalDir)) {
  throw new IllegalStateException("Refusing -atomic commit, final path exists: " + finalDir);
}

try {
  distCp.execute();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Target-path can't be committed")) {
    // copied data is safe in the temp dir named in the message; keep or replace target deliberately
  }
  throw e;
}

When it happens

Trigger: Running 'hadoop distcp -atomic <src> <dest>' where <dest> already exists at commit time: a previous atomic distcp already populated the target, another user or process created the target directory while the job ran, or a retried job re-committed after the first attempt had actually succeeded.

Common situations: Re-running an atomic distcp without clearing the previous target; concurrent writers racing to create the destination; expecting -atomic combined with -update/-overwrite to replace an existing final path (atomic commit never replaces an existing final dir); target left behind by an earlier canceled job.

Related errors

• Atomic commit failed. Temporary data in
• doesn't support renameSnapshot
• Fail to rename tmp file
• Failed to move block file for
• Failed to move meta file for

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d680112eec4cde46. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:602

        .build();
    DistCpContext distCpContext = new DistCpContext(options);
    distCpContext.setTargetPathExists(targetPathExists);

    target.buildListing(targetListing, distCpContext);
    DistCpUtils.sortListing(conf, targetListing, sortedTargetListing);
    return targetFinalPath;
  }

  private void commitData(Configuration conf) throws IOException {

    Path workDir = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH));
    Path finalDir = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH));
    FileSystem targetFS = workDir.getFileSystem(conf);

    LOG.info("Atomic commit enabled. Moving " + workDir + " to " + finalDir);
    if (targetFS.exists(finalDir) && targetFS.exists(workDir)) {
      LOG.error("Pre-existing final-path found at: " + finalDir);
      throw new IOException("Target-path can't be committed to because it " +
          "exists at " + finalDir + ". Copied data is in temp-dir: " + workDir + ". ");
    }

    boolean result = targetFS.rename(workDir, finalDir);
    if (!result) {
      LOG.warn("Rename failed. Perhaps data already moved. Verifying...");
      result = targetFS.exists(finalDir) && !targetFS.exists(workDir);
    }
    if (result) {
      LOG.info("Data committed successfully to " + finalDir);
      taskAttemptContext.setStatus("Data committed successfully to " + finalDir);
    } else {
      LOG.error("Unable to commit data to " + finalDir);
      throw new IOException("Atomic commit failed. Temporary data in " + workDir +
        ", Unable to move to " + finalDir);
    }
  }

View on GitHub (pinned to 2add963021)