apache/hadoop · error · InvalidInputException

Target path for atomic-commit already exists: {targetPath}.

Error message

Target path for atomic-commit already exists: {targetPath}. Cannot atomic-commit to pre-existing target-path.

What it means

-atomic works by copying into a temporary working directory and renaming it onto the target in one step; a rename cannot overlay an existing path, so a pre-existing target is rejected during listing. Even -overwrite does not bypass this check - atomic commit and an existing target are fundamentally incompatible.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java:174

            startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME);

    //If target is a file, then source has to be single file
    if (targetIsFile) {
      if (context.getSourcePaths().size() > 1) {
        throw new InvalidInputException("Multiple source being copied to a file: " +
            targetPath);
      }

      Path srcPath = context.getSourcePaths().get(0);
      FileSystem sourceFS = srcPath.getFileSystem(getConf());
      if (!sourceFS.isFile(srcPath)) {
        throw new InvalidInputException("Cannot copy " + srcPath +
            ", which is not a file to " + targetPath);
      }
    }

    if (context.shouldAtomicCommit() && targetExists) {
      throw new InvalidInputException("Target path for atomic-commit already exists: " +
        targetPath + ". Cannot atomic-commit to pre-existing target-path.");
    }

    for (Path path: context.getSourcePaths()) {
      FileSystem fs = path.getFileSystem(getConf());
      if (!fs.exists(path)) {
        throw new InvalidInputException(path + " doesn't exist");
      }
      if (Path.getPathWithoutSchemeAndAuthority(path).toString().
          startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME)) {
        if (!targetIsReservedRaw) {
          final String msg = "The source path '" + path + "' starts with " +
              HDFS_RESERVED_RAW_DIRECTORY_NAME + " but the target path '" +
              targetPath + "' does not. Either all or none of the paths must " +
              "have this prefix.";
          throw new InvalidInputException(msg);
        }
      } else if (targetIsReservedRaw) {

View on GitHub (pinned to 2add963021)

Solutions

Delete or rename the existing target, then re-run with -atomic: hadoop fs -rm -r <target>.
If the target must be preserved, drop -atomic and use a plain or -update copy instead.
For repeatable syncs into a fixed path, script 'rm -r target && distcp -atomic ...' or use -update without -atomic.

Example fix

# before: target already exists
hadoop distcp -atomic -overwrite hdfs://a/src hdfs://b/dest

# after: atomic commit needs a nonexistent target
hadoop fs -rm -r hdfs://b/dest
hadoop distcp -atomic hdfs://a/src hdfs://b/dest

Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight: -atomic requires a nonexistent target
if (context.shouldAtomicCommit()) {
  FileSystem tfs = context.getTargetPath().getFileSystem(conf);
  if (tfs.exists(context.getTargetPath())) {
    throw new InvalidInputException(
        "Atomic commit target already exists: " + context.getTargetPath());
  }
}

Try / catch

try {
  copyListing.doBuildListing(listingFile, context);
} catch (InvalidInputException e) {
  if (e.getMessage().startsWith("Target path for atomic-commit already exists")) {
    // delete/rename the target, or drop -atomic; -overwrite will NOT help
    handleAtomicTargetConflict(context.getTargetPath());
  } else {
    throw e;
  }
}

Prevention

Never combine -atomic with an expected-existing target; -overwrite does not bypass the check.
Script repeatable atomic copies as rm -r target followed by distcp -atomic.
Ensure no probe/tooling pre-creates the target directory.

When it happens

Trigger: hadoop distcp -atomic hdfs://a/src hdfs://b/dest when hdfs://b/dest already exists; re-running an atomic copy after a partially successful first run created the target.

Common situations: assuming -overwrite + -atomic is a valid combination; probe/tooling that pre-creates the target directory; retry loops that ignore this failure mode.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6602f534bd5538ed. Report an issue: GitHub.