{"record":{"id":"2ad736f63ea87721","repo":"apache/hadoop","slug":"the-tmp-directory-tmp-already-exists","errorCode":null,"errorMessage":"The tmp directory {tmp} already exists","messagePattern":"The tmp directory (.+?) already exists","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java","lineNumber":350,"sourceCode":"  private String getSnapshotName(String name) {\n    return Path.CUR_DIR.equals(name) ? \"\" : name;\n  }\n\n  private Path getSnapshotPath(Path inputDir, String snapshotName) {\n    if (Path.CUR_DIR.equals(snapshotName)) {\n      return inputDir;\n    } else {\n      return new Path(inputDir,\n          HdfsConstants.DOT_SNAPSHOT_DIR + Path.SEPARATOR + snapshotName);\n    }\n  }\n\n  private Path createTargetTmpDir(FileSystem targetFs,\n                                  Path targetDir) throws IOException {\n    final Path tmp = new Path(targetDir,\n        DistCpConstants.HDFS_DISTCP_DIFF_DIRECTORY_NAME + DistCp.rand.nextInt());\n    if (!targetFs.mkdirs(tmp)) {\n      throw new IOException(\"The tmp directory \" + tmp + \" already exists\");\n    }\n    return tmp;\n  }\n\n  private void deleteTargetTmpDir(FileSystem targetFs,\n                                  Path tmpDir) {\n    try {\n      if (tmpDir != null) {\n        targetFs.delete(tmpDir, true);\n      }\n    } catch (IOException e) {\n      DistCp.LOG.error(\"Unable to cleanup tmp dir: \" + tmpDir, e);\n    }\n  }\n\n  /**\n   * Compute the snapshot diff on the given file system. Return true if the diff\n   * is empty, i.e., no changes have happened in the FS.","sourceCodeStart":332,"sourceCodeEnd":368,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java#L332-L368","documentation":"Thrown by DistCp's incremental sync mode (-update -sync, driven by snapshot diffs) when it cannot create its staging directory. DistCpSync.createTargetTmpDir builds a path <targetDir>/.distcp.diff.tmp.<randomInt> and calls FileSystem.mkdirs(); a false return is reported as 'already exists'. Because the suffix is random, a genuine name collision is nearly impossible - in practice mkdirs() returned false because the target FileSystem refused or failed the creation (permissions, object-store semantics, raw/encryption restrictions) while returning false instead of throwing.","triggerScenarios":"Running 'hadoop distcp -update -sync [-diff s1,s2] hdfs://src hdfs://tgt' where targetFs.mkdirs(tgt/.distcp.diff.tmp.<n>) returns false: target directory not writable by the job user, an object-store FileSystem (s3a/abfs/gs) with non-POSIX directory semantics, two concurrent sync jobs that happened to draw the same DistCp.rand.nextInt(), or restriction policies (raw zone, encryption zone, router quotas) on the target directory.","commonSituations":"-sync or -diff jobs pointed at S3A/ABFS targets instead of HDFS; target dir owned by another user (no write permission); a previous sync run killed mid-flight leaking .distcp.diff.tmp.* directories that collide with a reused seed; RBAC or encryption-zone policies that silently deny directory creation.","solutions":["Re-run the distcp command: each attempt draws a fresh random suffix, so a transient or coincidental mkdirs() false clears itself.","Verify the job user can create directories under the target: hadoop fs -mkdir <targetDir>/.probe && hadoop fs -rm -r <targetDir>/.probe; fix ownership/permissions if this fails.","Clean leaked staging directories from killed prior runs: hadoop fs -rm -r '<targetDir>/.distcp.diff.tmp.*' (deleteTargetTmpDir runs in a finally block, but a killed JVM can leave them behind).","If the target is an object store, point the sync at an HDFS target - snapshot-diff sync relies on rename/concat semantics object stores do not honor.","Serialize concurrent distcp -sync jobs writing the same targetDir so staging directories never overlap."],"exampleFix":"# before: sync repeatedly fails even though the path is new\nhadoop distcp -update -sync hdfs://nn1/src hdfs://nn2/tgt\n\n# after: clean leaked staging dirs, confirm writability, retry\nhadoop fs -rm -r 'hdfs://nn2/tgt/.distcp.diff.tmp.*'\nhadoop fs -mkdir hdfs://nn2/tgt/.probe && hadoop fs -rm -r hdfs://nn2/tgt/.probe\nhadoop distcp -update -sync hdfs://nn1/src hdfs://nn2/tgt","handlingStrategy":"retry","validationCode":"// Before DistCp -sync: confirm the target can host the staging dir\nFileSystem tfs = targetDir.getFileSystem(conf);\nif (!tfs.isDirectory(targetDir)) {\n  throw new IllegalStateException(\"Target dir missing: \" + targetDir);\n}\nPath probe = new Path(targetDir, \".distcp.diff.tmp.probe\" + System.nanoTime());\nif (!tfs.mkdirs(probe)) {\n  throw new IllegalStateException(\"Cannot create tmp dir under \" + targetDir\n      + \" - check permissions / FS support\");\n}\ntfs.delete(probe, true);\n// best-effort cleanup of staging dirs leaked by killed runs\nFileStatus[] leaked = tfs.globStatus(new Path(targetDir, \".distcp.diff.tmp.*\"));\nif (leaked != null) {\n  for (FileStatus s : leaked) { tfs.delete(s.getPath(), true); }\n}","typeGuard":null,"tryCatchPattern":"try {\n  int exit = ToolRunner.run(conf, new DistCp(conf, options), args);\n} catch (IOException e) {\n  String m = e.getMessage();\n  if (m != null && m.startsWith(\"The tmp directory\") && m.endsWith(\"already exists\")) {\n    // random suffix: re-invocation draws a new name and almost always succeeds\n    LOG.warn(\"DistCp sync staging dir clash; retrying once\", e);\n    exit = ToolRunner.run(conf, new DistCp(conf, options), args); // fresh DistCp.rand draw\n  } else {\n    throw e;\n  }\n}","preventionTips":["Run at most one distcp -sync against the same target directory at a time.","Ensure the job user has write permission on the target directory before launching.","Do not use -sync/-diff against object stores; snapshot-diff sync assumes HDFS rename/concat semantics.","After killed sync jobs, clean up '<targetDir>/.distcp.diff.tmp.*' leftovers."],"tags":["distcp","hdfs","sync","snapshot-diff","mkdirs","staging-directory"],"backgroundTag":"tmp-directory-already-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}