apache/hadoop · error · IOException

Target-path can't be committed to because it exists at {fina

Error message

Target-path can't be committed to because it exists at {finalDir}. Copied data is in temp-dir: {workDir}.

What it means

Thrown by DistCp's CopyCommitter during the final atomic-commit step (hadoop distcp -atomic). commitData() reads the temp dir from distcp.target.work.path and the destination from distcp.target.final.path; before renaming work onto final it checks targetFS.exists(finalDir), and when the final path already exists alongside the work dir it aborts the commit so pre-existing data is never silently overwritten. The freshly copied data stays safe in the temp work directory named in the message.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:602

        .build();
    DistCpContext distCpContext = new DistCpContext(options);
    distCpContext.setTargetPathExists(targetPathExists);

    target.buildListing(targetListing, distCpContext);
    DistCpUtils.sortListing(conf, targetListing, sortedTargetListing);
    return targetFinalPath;
  }

  private void commitData(Configuration conf) throws IOException {

    Path workDir = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH));
    Path finalDir = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH));
    FileSystem targetFS = workDir.getFileSystem(conf);

    LOG.info("Atomic commit enabled. Moving " + workDir + " to " + finalDir);
    if (targetFS.exists(finalDir) && targetFS.exists(workDir)) {
      LOG.error("Pre-existing final-path found at: " + finalDir);
      throw new IOException("Target-path can't be committed to because it " +
          "exists at " + finalDir + ". Copied data is in temp-dir: " + workDir + ". ");
    }

    boolean result = targetFS.rename(workDir, finalDir);
    if (!result) {
      LOG.warn("Rename failed. Perhaps data already moved. Verifying...");
      result = targetFS.exists(finalDir) && !targetFS.exists(workDir);
    }
    if (result) {
      LOG.info("Data committed successfully to " + finalDir);
      taskAttemptContext.setStatus("Data committed successfully to " + finalDir);
    } else {
      LOG.error("Unable to commit data to " + finalDir);
      throw new IOException("Atomic commit failed. Temporary data in " + workDir +
        ", Unable to move to " + finalDir);
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

Move or delete the existing final directory (hdfs dfs -mv <finalDir> <finalDir>.bak) and re-run the distcp job
If the existing finalDir holds stale data, delete it and recover the new copy from the temp work dir printed in the error (hdfs dfs -mv <workDir> <finalDir>)
Verify nothing else creates the target while the job runs; aim -atomic runs at a fresh destination path
If replacing an existing target is the goal, drop -atomic and use -overwrite instead

Example fix

# before: /data/final already exists, so commitData() aborts
hadoop distcp -atomic hdfs://nn/src hdfs://nn/data/final

# after: move the old target aside, then re-run
hdfs dfs -mv hdfs://nn/data/final hdfs://nn/data/final.bak
hadoop distcp -atomic hdfs://nn/src hdfs://nn/data/final

Defensive patterns

Strategy: validation

Validate before calling

// before launching an atomic distcp, assert the final path is free
FileSystem tfs = finalDir.getFileSystem(conf);
if (tfs.exists(finalDir)) {
  throw new IllegalStateException("Refusing -atomic commit, final path exists: " + finalDir);
}

Try / catch

try {
  distCp.execute();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Target-path can't be committed")) {
    // copied data is safe in the temp dir named in the message; keep or replace target deliberately
  }
  throw e;
}

Prevention

Always aim -atomic at a fresh or pre-cleaned destination path
Archive the previous target (mv to .bak) as part of the runbook before re-running
Ensure a single writer owns the destination for the whole job lifetime
Never assume -update/-overwrite lets -atomic replace an existing final path

When it happens

Trigger: Running 'hadoop distcp -atomic <src> <dest>' where <dest> already exists at commit time: a previous atomic distcp already populated the target, another user or process created the target directory while the job ran, or a retried job re-committed after the first attempt had actually succeeded.

Common situations: Re-running an atomic distcp without clearing the previous target; concurrent writers racing to create the destination; expecting -atomic combined with -update/-overwrite to replace an existing final path (atomic commit never replaces an existing final dir); target left behind by an earlier canceled job.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d680112eec4cde46. Report an issue: GitHub.