{"record":{"id":"d680112eec4cde46","repo":"apache/hadoop","slug":"target-path-can-t-be-committed-to-because-it-exist","errorCode":null,"errorMessage":"Target-path can't be committed to because it exists at {finalDir}. Copied data is in temp-dir: {workDir}. ","messagePattern":"Target-path can't be committed to because it exists at (.+?)\\. Copied data is in temp-dir: (.+?)\\. ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java","lineNumber":602,"sourceCode":"        .build();\n    DistCpContext distCpContext = new DistCpContext(options);\n    distCpContext.setTargetPathExists(targetPathExists);\n\n    target.buildListing(targetListing, distCpContext);\n    DistCpUtils.sortListing(conf, targetListing, sortedTargetListing);\n    return targetFinalPath;\n  }\n\n  private void commitData(Configuration conf) throws IOException {\n\n    Path workDir = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_WORK_PATH));\n    Path finalDir = new Path(conf.get(DistCpConstants.CONF_LABEL_TARGET_FINAL_PATH));\n    FileSystem targetFS = workDir.getFileSystem(conf);\n\n    LOG.info(\"Atomic commit enabled. Moving \" + workDir + \" to \" + finalDir);\n    if (targetFS.exists(finalDir) && targetFS.exists(workDir)) {\n      LOG.error(\"Pre-existing final-path found at: \" + finalDir);\n      throw new IOException(\"Target-path can't be committed to because it \" +\n          \"exists at \" + finalDir + \". Copied data is in temp-dir: \" + workDir + \". \");\n    }\n\n    boolean result = targetFS.rename(workDir, finalDir);\n    if (!result) {\n      LOG.warn(\"Rename failed. Perhaps data already moved. Verifying...\");\n      result = targetFS.exists(finalDir) && !targetFS.exists(workDir);\n    }\n    if (result) {\n      LOG.info(\"Data committed successfully to \" + finalDir);\n      taskAttemptContext.setStatus(\"Data committed successfully to \" + finalDir);\n    } else {\n      LOG.error(\"Unable to commit data to \" + finalDir);\n      throw new IOException(\"Atomic commit failed. Temporary data in \" + workDir +\n        \", Unable to move to \" + finalDir);\n    }\n  }\n","sourceCodeStart":584,"sourceCodeEnd":620,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java#L584-L620","documentation":"Thrown by DistCp's CopyCommitter during the final atomic-commit step (hadoop distcp -atomic). commitData() reads the temp dir from distcp.target.work.path and the destination from distcp.target.final.path; before renaming work onto final it checks targetFS.exists(finalDir), and when the final path already exists alongside the work dir it aborts the commit so pre-existing data is never silently overwritten. The freshly copied data stays safe in the temp work directory named in the message.","triggerScenarios":"Running 'hadoop distcp -atomic <src> <dest>' where <dest> already exists at commit time: a previous atomic distcp already populated the target, another user or process created the target directory while the job ran, or a retried job re-committed after the first attempt had actually succeeded.","commonSituations":"Re-running an atomic distcp without clearing the previous target; concurrent writers racing to create the destination; expecting -atomic combined with -update/-overwrite to replace an existing final path (atomic commit never replaces an existing final dir); target left behind by an earlier canceled job.","solutions":["Move or delete the existing final directory (hdfs dfs -mv <finalDir> <finalDir>.bak) and re-run the distcp job","If the existing finalDir holds stale data, delete it and recover the new copy from the temp work dir printed in the error (hdfs dfs -mv <workDir> <finalDir>)","Verify nothing else creates the target while the job runs; aim -atomic runs at a fresh destination path","If replacing an existing target is the goal, drop -atomic and use -overwrite instead"],"exampleFix":"# before: /data/final already exists, so commitData() aborts\nhadoop distcp -atomic hdfs://nn/src hdfs://nn/data/final\n\n# after: move the old target aside, then re-run\nhdfs dfs -mv hdfs://nn/data/final hdfs://nn/data/final.bak\nhadoop distcp -atomic hdfs://nn/src hdfs://nn/data/final","handlingStrategy":"validation","validationCode":"// before launching an atomic distcp, assert the final path is free\nFileSystem tfs = finalDir.getFileSystem(conf);\nif (tfs.exists(finalDir)) {\n  throw new IllegalStateException(\"Refusing -atomic commit, final path exists: \" + finalDir);\n}","typeGuard":null,"tryCatchPattern":"try {\n  distCp.execute();\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Target-path can't be committed\")) {\n    // copied data is safe in the temp dir named in the message; keep or replace target deliberately\n  }\n  throw e;\n}","preventionTips":["Always aim -atomic at a fresh or pre-cleaned destination path","Archive the previous target (mv to .bak) as part of the runbook before re-running","Ensure a single writer owns the destination for the whole job lifetime","Never assume -update/-overwrite lets -atomic replace an existing final path"],"tags":["distcp","hadoop","atomic-commit","rename","hdfs"],"backgroundTag":"atomic-commit-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}