{"record":{"id":"59fd364da9b3e25b","repo":"apache/hadoop","slug":"atomic-commit-failed-temporary-data-in-workdir","errorCode":null,"errorMessage":"Atomic commit failed. Temporary data in {workDir}, Unable to move to {finalDir}","messagePattern":"Atomic commit failed\\. Temporary data in (.+?), Unable to move to (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java","lineNumber":616,"sourceCode":"\n    LOG.info(\"Atomic commit enabled. Moving \" + workDir + \" to \" + finalDir);\n    if (targetFS.exists(finalDir) && targetFS.exists(workDir)) {\n      LOG.error(\"Pre-existing final-path found at: \" + finalDir);\n      throw new IOException(\"Target-path can't be committed to because it \" +\n          \"exists at \" + finalDir + \". Copied data is in temp-dir: \" + workDir + \". \");\n    }\n\n    boolean result = targetFS.rename(workDir, finalDir);\n    if (!result) {\n      LOG.warn(\"Rename failed. Perhaps data already moved. Verifying...\");\n      result = targetFS.exists(finalDir) && !targetFS.exists(workDir);\n    }\n    if (result) {\n      LOG.info(\"Data committed successfully to \" + finalDir);\n      taskAttemptContext.setStatus(\"Data committed successfully to \" + finalDir);\n    } else {\n      LOG.error(\"Unable to commit data to \" + finalDir);\n      throw new IOException(\"Atomic commit failed. Temporary data in \" + workDir +\n        \", Unable to move to \" + finalDir);\n    }\n  }\n\n  /**\n   * Concat the passed chunk files into one and rename it the targetFile.\n   */\n  private void concatFileChunks(Configuration conf, Path sourceFile,\n                                Path targetFile, LinkedList<Path> allChunkPaths,\n                                CopyListingFileStatus srcFileStatus)\n      throws IOException {\n    if (allChunkPaths.size() == 1) {\n      return;\n    }\n    if (LOG.isDebugEnabled()) {\n      LOG.debug(\"concat \" + targetFile + \" allChunkSize+ \"\n          + allChunkPaths.size());\n    }","sourceCodeStart":598,"sourceCodeEnd":634,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java#L598-L634","documentation":"Final step of DistCp's atomic commit: targetFS.rename(workDir, finalDir) returned false, and the fallback verification (finalDir exists AND workDir no longer exists) also failed, so CopyCommitter cannot complete the -atomic commit and throws. The copied data is still intact in the work directory named in the message; the target FileSystem itself rejected the rename.","triggerScenarios":"rename() attempted across namespaces (work and final paths resolve to different clusters/NameNodes), missing or non-writable parent of finalDir, NameNode in safe mode, HA failover invalidating the handle, quota exhaustion, or the work dir deleted externally mid-commit.","commonSituations":"Unqualified or mixed-URI target paths putting work and final dirs on different filesystems; target parent deleted or its permissions changed while the job ran; safe mode after a NameNode restart; quota or disk full on the target.","solutions":["Check NameNode logs for the rejected rename and fix the root cause (safe mode: hdfs dfsadmin -safemode leave; permissions: chmod/chown on the finalDir parent)","Once the filesystem is healthy, finish the commit by hand using the paths from the error: hdfs dfs -mv <workDir> <finalDir>","Use fully-qualified URIs (hdfs://nameservice/...) so the -atomic work and final paths resolve to the same filesystem","Re-run with -update after committing or discarding the temp data so already-copied files are skipped"],"exampleFix":"# before: atomic commit failed, data stranded in the temp dir\n# (rename rejected: safe mode / missing parent / cross-namespace)\n\n# after: fix the blocker, then complete the commit manually\nhdfs dfsadmin -safemode leave\nhdfs dfs -mkdir -p /data\nhdfs dfs -mv <workDir-from-error> <finalDir-from-error>","handlingStrategy":"try-catch","validationCode":"// atomic commit needs a same-filesystem rename; verify before submit\nFileSystem fsWork = workDir.getFileSystem(conf);\nFileSystem fsFinal = finalDir.getFileSystem(conf);\nif (!fsWork.getUri().equals(fsFinal.getUri())\n    || finalDir.getParent() == null\n    || !fsFinal.exists(finalDir.getParent())) {\n  throw new IllegalStateException(\"work/final dirs not on one filesystem or final parent missing\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(true);\n} catch (IOException e) {\n  // record the workDir for recovery; after fixing the FS issue run:\n  // hdfs dfs -mv <workDir> <finalDir>\n  LOG.error(\"atomic commit failed, temp data at {}\", workDir, e);\n}","preventionTips":["Use fully-qualified URIs so work and final paths resolve to one filesystem","Keep the final dir's parent present and writable for the job duration","Monitor the target NameNode for safe mode during long copies","Keep the temp work-dir path in logs so a manual mv can finish the commit"],"tags":["distcp","hadoop","atomic-commit","rename","safe-mode","hdfs"],"backgroundTag":"rename-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}