{"record":{"id":"2e6a87bf244fb186","repo":"apache/hadoop","slug":"work-path-workdir-and-target-path-targ","errorCode":null,"errorMessage":"Work path \" + workDir + \" and target path \" + targetPath + \" are in different file system","messagePattern":"Work path \" \\+ workDir \\+ \" and target path \" \\+ targetPath \\+ \" are in different file system","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java","lineNumber":365,"sourceCode":"                                          targetFS.getWorkingDirectory());\n    if (context.shouldPreserve(\n        DistCpOptions.FileAttribute.ACL)) {\n      DistCpUtils.checkFileSystemAclSupport(targetFS);\n    }\n    if (context.shouldPreserve(\n        DistCpOptions.FileAttribute.XATTR)) {\n      DistCpUtils.checkFileSystemXAttrSupport(targetFS);\n    }\n    if (context.shouldAtomicCommit()) {\n      Path workDir = context.getAtomicWorkPath();\n      if (workDir == null) {\n        workDir = targetPath.getParent();\n      }\n      workDir = new Path(workDir, WIP_PREFIX + targetPath.getName()\n                                + rand.nextInt());\n      FileSystem workFS = workDir.getFileSystem(configuration);\n      if (!FileUtil.compareFs(targetFS, workFS)) {\n        throw new IllegalArgumentException(\"Work path \" + workDir +\n            \" and target path \" + targetPath + \" are in different file system\");\n      }\n      CopyOutputFormat.setWorkingDirectory(job, workDir);\n    } else {\n      CopyOutputFormat.setWorkingDirectory(job, targetPath);\n    }\n    CopyOutputFormat.setCommitDirectory(job, targetPath);\n\n    Path logPath = context.getLogPath();\n    if (logPath == null) {\n      logPath = new Path(metaFolder, \"_logs\");\n    } else {\n      LOG.info(\"DistCp job log path: \" + logPath);\n    }\n    CopyOutputFormat.setOutputPath(job, logPath);\n  }\n\n  /**","sourceCodeStart":347,"sourceCodeEnd":383,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java#L347-L383","documentation":"With -atomic, DistCp stages files in a work directory (from -tmp, else the target's parent) and commits by rename, which is only a metadata operation within one filesystem. It therefore checks FileUtil.compareFs(targetFS, workFS); different scheme or authority throws IllegalArgumentException('Work path X and target path Y are in different file system') before the job is submitted.","triggerScenarios":"hadoop distcp -atomic -tmp file:///scratch ... hdfs://nn/dst (local tmp, HDFS target); or -tmp hdfs://clusterA/tmp with the target on hdfs://clusterB/dst (same scheme, different authority); HA nameservice aliases spelled differently between tmp and target URIs.","commonSituations":"Cross-namespace HDFS-to-HDFS copies where the default tmp resolves to the wrong cluster; runbooks written against local testing reused on cluster URIs; mixed authority spellings (nameservice vs..rpcAddress) for the same cluster.","solutions":["Set -tmp on the same filesystem and authority as the target: -tmp hdfs://nnB/tmp/distcp for target hdfs://nnB/dst","Or omit -tmp so the work path defaults to the target's parent (same FS by construction)","Spell the tmp and target URIs with the identical scheme/authority (same HA nameservice)","Or drop -atomic, accepting non-atomic output, when staging on the target FS is not possible"],"exampleFix":"# before\nhadoop distcp -atomic -tmp file:///scratch/stage hdfs://nnA/src hdfs://nnB/dst\n# -> Work path file:///scratch/stage and target path hdfs://nnB/dst are in different file system\n\n# after\nhadoop distcp -atomic -tmp hdfs://nnB/tmp/distcp-stage hdfs://nnA/src hdfs://nnB/dst","handlingStrategy":"validation","validationCode":"FileSystem targetFS = targetPath.getFileSystem(conf);\nPath workPath = (tmpPath != null) ? tmpPath : targetPath.getParent();\nFileSystem workFS = workPath.getFileSystem(conf);\nif (!FileUtil.compareFs(targetFS, workFS)) {\n  throw new IllegalArgumentException(\n      \"atomic work path \" + workPath + \" must live on \" + targetFS.getUri());\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["When generating distcp commands programmatically, derive -tmp from the target URI rather than a global scratch setting","For cross-cluster copies, always stage on the destination cluster","Use identical authority spelling (HA nameservice) in tmp and target URIs"],"tags":["distcp","atomic-commit","filesystem-mismatch","tmp-directory"],"backgroundTag":"cross-filesystem-path-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}