{"record":{"id":"6602f534bd5538ed","repo":"apache/hadoop","slug":"target-path-for-atomic-commit-already-exists-tar","errorCode":null,"errorMessage":"Target path for atomic-commit already exists: {targetPath}. Cannot atomic-commit to pre-existing target-path.","messagePattern":"Target path for atomic-commit already exists: (.+?)\\. Cannot atomic-commit to pre-existing target-path\\.","errorType":"exception","errorClass":"InvalidInputException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java","lineNumber":174,"sourceCode":"            startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME);\n\n    //If target is a file, then source has to be single file\n    if (targetIsFile) {\n      if (context.getSourcePaths().size() > 1) {\n        throw new InvalidInputException(\"Multiple source being copied to a file: \" +\n            targetPath);\n      }\n\n      Path srcPath = context.getSourcePaths().get(0);\n      FileSystem sourceFS = srcPath.getFileSystem(getConf());\n      if (!sourceFS.isFile(srcPath)) {\n        throw new InvalidInputException(\"Cannot copy \" + srcPath +\n            \", which is not a file to \" + targetPath);\n      }\n    }\n\n    if (context.shouldAtomicCommit() && targetExists) {\n      throw new InvalidInputException(\"Target path for atomic-commit already exists: \" +\n        targetPath + \". Cannot atomic-commit to pre-existing target-path.\");\n    }\n\n    for (Path path: context.getSourcePaths()) {\n      FileSystem fs = path.getFileSystem(getConf());\n      if (!fs.exists(path)) {\n        throw new InvalidInputException(path + \" doesn't exist\");\n      }\n      if (Path.getPathWithoutSchemeAndAuthority(path).toString().\n          startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME)) {\n        if (!targetIsReservedRaw) {\n          final String msg = \"The source path '\" + path + \"' starts with \" +\n              HDFS_RESERVED_RAW_DIRECTORY_NAME + \" but the target path '\" +\n              targetPath + \"' does not. Either all or none of the paths must \" +\n              \"have this prefix.\";\n          throw new InvalidInputException(msg);\n        }\n      } else if (targetIsReservedRaw) {","sourceCodeStart":156,"sourceCodeEnd":192,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L156-L192","documentation":"-atomic works by copying into a temporary working directory and renaming it onto the target in one step; a rename cannot overlay an existing path, so a pre-existing target is rejected during listing. Even -overwrite does not bypass this check - atomic commit and an existing target are fundamentally incompatible.","triggerScenarios":"hadoop distcp -atomic hdfs://a/src hdfs://b/dest when hdfs://b/dest already exists; re-running an atomic copy after a partially successful first run created the target.","commonSituations":"assuming -overwrite + -atomic is a valid combination; probe/tooling that pre-creates the target directory; retry loops that ignore this failure mode.","solutions":["Delete or rename the existing target, then re-run with -atomic: hadoop fs -rm -r <target>.","If the target must be preserved, drop -atomic and use a plain or -update copy instead.","For repeatable syncs into a fixed path, script 'rm -r target && distcp -atomic ...' or use -update without -atomic."],"exampleFix":"# before: target already exists\nhadoop distcp -atomic -overwrite hdfs://a/src hdfs://b/dest\n\n# after: atomic commit needs a nonexistent target\nhadoop fs -rm -r hdfs://b/dest\nhadoop distcp -atomic hdfs://a/src hdfs://b/dest","handlingStrategy":"validation","validationCode":"// Pre-flight: -atomic requires a nonexistent target\nif (context.shouldAtomicCommit()) {\n  FileSystem tfs = context.getTargetPath().getFileSystem(conf);\n  if (tfs.exists(context.getTargetPath())) {\n    throw new InvalidInputException(\n        \"Atomic commit target already exists: \" + context.getTargetPath());\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  copyListing.doBuildListing(listingFile, context);\n} catch (InvalidInputException e) {\n  if (e.getMessage().startsWith(\"Target path for atomic-commit already exists\")) {\n    // delete/rename the target, or drop -atomic; -overwrite will NOT help\n    handleAtomicTargetConflict(context.getTargetPath());\n  } else {\n    throw e;\n  }\n}","preventionTips":["Never combine -atomic with an expected-existing target; -overwrite does not bypass the check.","Script repeatable atomic copies as rm -r target followed by distcp -atomic.","Ensure no probe/tooling pre-creates the target directory."],"tags":["distcp","atomic-commit","target-path","input-validation"],"backgroundTag":"atomic-commit-target-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}