apache/hadoop · error · IOException

The tmp directory {tmp} already exists

Error message

The tmp directory {tmp} already exists

What it means

Thrown by DistCp's incremental sync mode (-update -sync, driven by snapshot diffs) when it cannot create its staging directory. DistCpSync.createTargetTmpDir builds a path <targetDir>/.distcp.diff.tmp.<randomInt> and calls FileSystem.mkdirs(); a false return is reported as 'already exists'. Because the suffix is random, a genuine name collision is nearly impossible - in practice mkdirs() returned false because the target FileSystem refused or failed the creation (permissions, object-store semantics, raw/encryption restrictions) while returning false instead of throwing.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java:350

  private String getSnapshotName(String name) {
    return Path.CUR_DIR.equals(name) ? "" : name;
  }

  private Path getSnapshotPath(Path inputDir, String snapshotName) {
    if (Path.CUR_DIR.equals(snapshotName)) {
      return inputDir;
    } else {
      return new Path(inputDir,
          HdfsConstants.DOT_SNAPSHOT_DIR + Path.SEPARATOR + snapshotName);
    }
  }

  private Path createTargetTmpDir(FileSystem targetFs,
                                  Path targetDir) throws IOException {
    final Path tmp = new Path(targetDir,
        DistCpConstants.HDFS_DISTCP_DIFF_DIRECTORY_NAME + DistCp.rand.nextInt());
    if (!targetFs.mkdirs(tmp)) {
      throw new IOException("The tmp directory " + tmp + " already exists");
    }
    return tmp;
  }

  private void deleteTargetTmpDir(FileSystem targetFs,
                                  Path tmpDir) {
    try {
      if (tmpDir != null) {
        targetFs.delete(tmpDir, true);
      }
    } catch (IOException e) {
      DistCp.LOG.error("Unable to cleanup tmp dir: " + tmpDir, e);
    }
  }

  /**
   * Compute the snapshot diff on the given file system. Return true if the diff
   * is empty, i.e., no changes have happened in the FS.

View on GitHub (pinned to 2add963021)

Solutions

1. Re-run the distcp command: each attempt draws a fresh random suffix, so a transient or coincidental mkdirs() false clears itself.
2. Verify the job user can create directories under the target: hadoop fs -mkdir <targetDir>/.probe && hadoop fs -rm -r <targetDir>/.probe; fix ownership/permissions if this fails.
3. Clean leaked staging directories from killed prior runs: hadoop fs -rm -r '<targetDir>/.distcp.diff.tmp.*' (deleteTargetTmpDir runs in a finally block, but a killed JVM can leave them behind).
4. If the target is an object store, point the sync at an HDFS target - snapshot-diff sync relies on rename/concat semantics object stores do not honor.
5. Serialize concurrent distcp -sync jobs writing the same targetDir so staging directories never overlap.

Example fix

# before: sync repeatedly fails even though the path is new
hadoop distcp -update -sync hdfs://nn1/src hdfs://nn2/tgt

# after: clean leaked staging dirs, confirm writability, retry
hadoop fs -rm -r 'hdfs://nn2/tgt/.distcp.diff.tmp.*'
hadoop fs -mkdir hdfs://nn2/tgt/.probe && hadoop fs -rm -r hdfs://nn2/tgt/.probe
hadoop distcp -update -sync hdfs://nn1/src hdfs://nn2/tgt

// Before DistCp -sync: confirm the target can host the staging dir
FileSystem tfs = targetDir.getFileSystem(conf);
if (!tfs.isDirectory(targetDir)) {
  throw new IllegalStateException("Target dir missing: " + targetDir);
}
Path probe = new Path(targetDir, ".distcp.diff.tmp.probe" + System.nanoTime());
if (!tfs.mkdirs(probe)) {
  throw new IllegalStateException("Cannot create tmp dir under " + targetDir
      + " - check permissions / FS support");
}
tfs.delete(probe, true);
// best-effort cleanup of staging dirs leaked by killed runs
FileStatus[] leaked = tfs.globStatus(new Path(targetDir, ".distcp.diff.tmp.*"));
if (leaked != null) {
  for (FileStatus s : leaked) { tfs.delete(s.getPath(), true); }
}

try {
  int exit = ToolRunner.run(conf, new DistCp(conf, options), args);
} catch (IOException e) {
  String m = e.getMessage();
  if (m != null && m.startsWith("The tmp directory") && m.endsWith("already exists")) {
    // random suffix: re-invocation draws a new name and almost always succeeds
    LOG.warn("DistCp sync staging dir clash; retrying once", e);
    exit = ToolRunner.run(conf, new DistCp(conf, options), args); // fresh DistCp.rand draw
  } else {
    throw e;
  }
}

When it happens

Trigger: Running 'hadoop distcp -update -sync [-diff s1,s2] hdfs://src hdfs://tgt' where targetFs.mkdirs(tgt/.distcp.diff.tmp.<n>) returns false: target directory not writable by the job user, an object-store FileSystem (s3a/abfs/gs) with non-POSIX directory semantics, two concurrent sync jobs that happened to draw the same DistCp.rand.nextInt(), or restriction policies (raw zone, encryption zone, router quotas) on the target directory.

Common situations: -sync or -diff jobs pointed at S3A/ABFS targets instead of HDFS; target dir owned by another user (no write permission); a previous sync run killed mid-flight leaking .distcp.diff.tmp.* directories that collide with a reused seed; RBAC or encryption-zone policies that silently deny directory creation.

Related errors

• Mkdirs failed to create directory {dirName}
• Cannot create directory {}
• Failed to mkdirs {}
• Mkdirs failed to create {}
• Mkdirs failed to create ${dir}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2ad736f63ea87721. Report an issue: GitHub.