apache/hadoop · error · IOException

Atomic commit failed. Temporary data in {workDir}, Unable to

Error message

Atomic commit failed. Temporary data in {workDir}, Unable to move to {finalDir}

What it means

Final step of DistCp's atomic commit: targetFS.rename(workDir, finalDir) returned false, and the fallback verification (finalDir exists AND workDir no longer exists) also failed, so CopyCommitter cannot complete the -atomic commit and throws. The copied data is still intact in the work directory named in the message; the target FileSystem itself rejected the rename.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java:616

    LOG.info("Atomic commit enabled. Moving " + workDir + " to " + finalDir);
    if (targetFS.exists(finalDir) && targetFS.exists(workDir)) {
      LOG.error("Pre-existing final-path found at: " + finalDir);
      throw new IOException("Target-path can't be committed to because it " +
          "exists at " + finalDir + ". Copied data is in temp-dir: " + workDir + ". ");
    }

    boolean result = targetFS.rename(workDir, finalDir);
    if (!result) {
      LOG.warn("Rename failed. Perhaps data already moved. Verifying...");
      result = targetFS.exists(finalDir) && !targetFS.exists(workDir);
    }
    if (result) {
      LOG.info("Data committed successfully to " + finalDir);
      taskAttemptContext.setStatus("Data committed successfully to " + finalDir);
    } else {
      LOG.error("Unable to commit data to " + finalDir);
      throw new IOException("Atomic commit failed. Temporary data in " + workDir +
        ", Unable to move to " + finalDir);
    }
  }

  /**
   * Concat the passed chunk files into one and rename it the targetFile.
   */
  private void concatFileChunks(Configuration conf, Path sourceFile,
                                Path targetFile, LinkedList<Path> allChunkPaths,
                                CopyListingFileStatus srcFileStatus)
      throws IOException {
    if (allChunkPaths.size() == 1) {
      return;
    }
    if (LOG.isDebugEnabled()) {
      LOG.debug("concat " + targetFile + " allChunkSize+ "
          + allChunkPaths.size());
    }

View on GitHub (pinned to 2add963021)

Solutions

Check NameNode logs for the rejected rename and fix the root cause (safe mode: hdfs dfsadmin -safemode leave; permissions: chmod/chown on the finalDir parent)
Once the filesystem is healthy, finish the commit by hand using the paths from the error: hdfs dfs -mv <workDir> <finalDir>
Use fully-qualified URIs (hdfs://nameservice/...) so the -atomic work and final paths resolve to the same filesystem
Re-run with -update after committing or discarding the temp data so already-copied files are skipped

Example fix

# before: atomic commit failed, data stranded in the temp dir
# (rename rejected: safe mode / missing parent / cross-namespace)

# after: fix the blocker, then complete the commit manually
hdfs dfsadmin -safemode leave
hdfs dfs -mkdir -p /data
hdfs dfs -mv <workDir-from-error> <finalDir-from-error>

Defensive patterns

Strategy: try-catch

Validate before calling

// atomic commit needs a same-filesystem rename; verify before submit
FileSystem fsWork = workDir.getFileSystem(conf);
FileSystem fsFinal = finalDir.getFileSystem(conf);
if (!fsWork.getUri().equals(fsFinal.getUri())
    || finalDir.getParent() == null
    || !fsFinal.exists(finalDir.getParent())) {
  throw new IllegalStateException("work/final dirs not on one filesystem or final parent missing");
}

Try / catch

try {
  job.waitForCompletion(true);
} catch (IOException e) {
  // record the workDir for recovery; after fixing the FS issue run:
  // hdfs dfs -mv <workDir> <finalDir>
  LOG.error("atomic commit failed, temp data at {}", workDir, e);
}

Prevention

Use fully-qualified URIs so work and final paths resolve to one filesystem
Keep the final dir's parent present and writable for the job duration
Monitor the target NameNode for safe mode during long copies
Keep the temp work-dir path in logs so a manual mv can finish the commit

When it happens

Trigger: rename() attempted across namespaces (work and final paths resolve to different clusters/NameNodes), missing or non-writable parent of finalDir, NameNode in safe mode, HA failover invalidating the handle, quota exhaustion, or the work dir deleted externally mid-commit.

Common situations: Unqualified or mixed-URI target paths putting work and final dirs on different filesystems; target parent deleted or its permissions changed while the job ran; safe mode after a NameNode restart; quota or disk full on the target.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/59fd364da9b3e25b. Report an issue: GitHub.