apache/hadoop · error · IllegalArgumentException

Work path " + workDir + " and target path " + targetPath + "

Error message

Work path " + workDir + " and target path " + targetPath + " are in different file system

What it means

With -atomic, DistCp stages files in a work directory (from -tmp, else the target's parent) and commits by rename, which is only a metadata operation within one filesystem. It therefore checks FileUtil.compareFs(targetFS, workFS); different scheme or authority throws IllegalArgumentException('Work path X and target path Y are in different file system') before the job is submitted.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java:365

                                          targetFS.getWorkingDirectory());
    if (context.shouldPreserve(
        DistCpOptions.FileAttribute.ACL)) {
      DistCpUtils.checkFileSystemAclSupport(targetFS);
    }
    if (context.shouldPreserve(
        DistCpOptions.FileAttribute.XATTR)) {
      DistCpUtils.checkFileSystemXAttrSupport(targetFS);
    }
    if (context.shouldAtomicCommit()) {
      Path workDir = context.getAtomicWorkPath();
      if (workDir == null) {
        workDir = targetPath.getParent();
      }
      workDir = new Path(workDir, WIP_PREFIX + targetPath.getName()
                                + rand.nextInt());
      FileSystem workFS = workDir.getFileSystem(configuration);
      if (!FileUtil.compareFs(targetFS, workFS)) {
        throw new IllegalArgumentException("Work path " + workDir +
            " and target path " + targetPath + " are in different file system");
      }
      CopyOutputFormat.setWorkingDirectory(job, workDir);
    } else {
      CopyOutputFormat.setWorkingDirectory(job, targetPath);
    }
    CopyOutputFormat.setCommitDirectory(job, targetPath);

    Path logPath = context.getLogPath();
    if (logPath == null) {
      logPath = new Path(metaFolder, "_logs");
    } else {
      LOG.info("DistCp job log path: " + logPath);
    }
    CopyOutputFormat.setOutputPath(job, logPath);
  }

  /**

View on GitHub (pinned to 2add963021)

Solutions

1. Set -tmp on the same filesystem and authority as the target: -tmp hdfs://nnB/tmp/distcp for target hdfs://nnB/dst
2. Or omit -tmp so the work path defaults to the target's parent (same FS by construction)
3. Spell the tmp and target URIs with the identical scheme/authority (same HA nameservice)
4. Or drop -atomic, accepting non-atomic output, when staging on the target FS is not possible

Example fix

# before
hadoop distcp -atomic -tmp file:///scratch/stage hdfs://nnA/src hdfs://nnB/dst
# -> Work path file:///scratch/stage and target path hdfs://nnB/dst are in different file system

# after
hadoop distcp -atomic -tmp hdfs://nnB/tmp/distcp-stage hdfs://nnA/src hdfs://nnB/dst

FileSystem targetFS = targetPath.getFileSystem(conf);
Path workPath = (tmpPath != null) ? tmpPath : targetPath.getParent();
FileSystem workFS = workPath.getFileSystem(conf);
if (!FileUtil.compareFs(targetFS, workFS)) {
  throw new IllegalArgumentException(
      "atomic work path " + workPath + " must live on " + targetFS.getUri());
}

When it happens

Trigger: hadoop distcp -atomic -tmp file:///scratch ... hdfs://nn/dst (local tmp, HDFS target); or -tmp hdfs://clusterA/tmp with the target on hdfs://clusterB/dst (same scheme, different authority); HA nameservice aliases spelled differently between tmp and target URIs.

Common situations: Cross-namespace HDFS-to-HDFS copies where the default tmp resolves to the wrong cluster; runbooks written against local testing reused on cluster URIs; mixed authority spellings (nameservice vs..rpcAddress) for the same cluster.

Related errors

• Target path for atomic-commit already exists: {targetPath}.
• Target-path can't be committed to because it exists at {fina
• Atomic commit failed. Temporary data in {workDir}, Unable to
• key + ": No such file or directory."
• Har: rename not allowed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2e6a87bf244fb186. Report an issue: GitHub.