{"record":{"id":"69d4a63b42ce1622","repo":"apache/hadoop","slug":"file-exists","errorCode":null,"errorMessage":"File exists","messagePattern":"File exists","errorType":"exception","errorClass":"PathExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java","lineNumber":238,"sourceCode":"    }\n  }\n\n  @Override\n  protected void processArguments(LinkedList<PathData> args)\n  throws IOException {\n    // if more than one arg, the destination must be a directory\n    // if one arg, the dst must not exist or must be a directory\n    if (args.size() > 1) {\n      if (!dst.exists) {\n        throw new PathNotFoundException(dst.toString());\n      }\n      if (!dst.stat.isDirectory()) {\n        throw new PathIsNotDirectoryException(dst.toString());\n      }\n    } else if (dst.exists) {\n      if (!dst.stat.isDirectory() && !overwrite) {\n        LOG.debug(\"Destination file exists: {}\", dst.stat);\n        throw new PathExistsException(dst.toString());\n      }\n    } else if (!dst.parentExists()) {\n      throw new PathNotFoundException(dst.toString())\n          .withFullyQualifiedPath(dst.path.toUri().toString());\n    }\n    super.processArguments(args);\n  }\n\n  @Override\n  protected void processPathArgument(PathData src)\n  throws IOException {\n    if (src.stat.isDirectory() && src.fs.equals(dst.fs)) {\n      PathData target = getTargetPath(src);\n      String srcPath = src.fs.makeQualified(src.path).toString();\n      String dstPath = dst.fs.makeQualified(target.path).toString();\n      if (dstPath.equals(srcPath)) {\n        PathIOException e = new PathIOException(src.toString(),\n            \"are identical\");","sourceCodeStart":220,"sourceCodeEnd":256,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CommandWithDestination.java#L220-L256","documentation":"Thrown by CommandWithDestination.processArguments() in the single-source case: the destination exists, is NOT a directory, and no -f (overwrite) flag was set. PathExistsException renders as 'dst: File exists'. The shell deliberately refuses to clobber an existing file without explicit consent.","triggerScenarios":"'hdfs dfs -put new.csv /warehouse/data.csv' when data.csv already exists and -f omitted; re-running a failed pipeline step whose output file was already written; -cp onto an existing file without -f.","commonSituations":"Job retries and backfill scripts hitting their own previous partial output; cron jobs assuming overwrite semantics like GNU cp -f; developers expecting put to truncate like '>' in shell redirection.","solutions":["Add the overwrite flag: 'hdfs dfs -put -f new.csv /warehouse/data.csv'","If the existing file must be kept, choose a new destination name (e.g. timestamped)","Delete or archive the old file first ('hdfs dfs -rm') when -f is unavailable for the command variant in use"],"exampleFix":"# before\nhdfs dfs -put new.csv /warehouse/data.csv\n# after\nhdfs dfs -put -f new.csv /warehouse/data.csv","handlingStrategy":"validation","validationCode":"// single-source copy: decide overwrite policy up front\nboolean overwriteWanted = true;\nif (fs.exists(dst) && !fs.getFileStatus(dst).isDirectory() && !overwriteWanted) {\n  throw new FileAlreadyExistsException(\"refusing to overwrite existing file \" + dst);\n}","typeGuard":null,"tryCatchPattern":"try {\n  shellRun(\"-put\", src, dst);\n} catch (PathExistsException e) {\n  // existing file at destination and no -f given; retry with explicit overwrite\n  shellRun(\"-put\", \"-f\", src, dst);\n}","preventionTips":["Pass -f whenever the pipeline's semantics are 'latest write wins'","Or write to unique names (jobId/timestamp) and rename atomically for exactly-once outputs","Treat 'File exists' as a signal of duplicate/retried work, not just a flag problem"],"tags":["hadoop","hdfs","file-exists","overwrite","copy","shell"],"backgroundTag":"file-already-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}