{"record":{"id":"6858499b1d9a7d1e","repo":"apache/hadoop","slug":"e-dest-exists","errorCode":"E_DEST_EXISTS","errorMessage":"output path is not a directory: Destination path exists and committer conflict resolution mode is \"fail\"","messagePattern":"output path is not a directory: Destination path exists and committer conflict resolution mode is \"fail\"","errorType":"exception","errorClass":"PathExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/DirectoryStagingCommitter.java","lineNumber":78,"sourceCode":"\n  @Override\n  public String getName() {\n    return NAME;\n  }\n\n  @Override\n  public void setupJob(JobContext context) throws IOException {\n    Path outputPath = getOutputPath();\n    FileSystem fs = getDestFS();\n    ConflictResolution conflictResolution = getConflictResolutionMode(\n        context, fs.getConf());\n    LOG.info(\"Conflict Resolution mode is {}\", conflictResolution);\n    try {\n      final FileStatus status = fs.getFileStatus(outputPath);\n\n      // if it is not a directory, fail fast for all conflict options.\n      if (!status.isDirectory()) {\n        throw new PathExistsException(outputPath.toString(),\n            \"output path is not a directory: \"\n                + InternalCommitterConstants.E_DEST_EXISTS);\n      }\n      switch(conflictResolution) {\n      case FAIL:\n        throw failDestinationExists(outputPath,\n            \"Setting job as \" + getRole());\n      case APPEND:\n      case REPLACE:\n        LOG.debug(\"Destination directory exists; conflict policy permits this\");\n      }\n    } catch (FileNotFoundException ignored) {\n      // there is no destination path, hence, no conflict.\n    }\n    // make the parent directory, which also triggers a recursive directory\n    // creation operation\n    super.setupJob(context);\n  }","sourceCodeStart":60,"sourceCodeEnd":96,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/DirectoryStagingCommitter.java#L60-L96","documentation":"PathExistsException from DirectoryStagingCommitter.setupJob when the destination path already exists but is NOT a directory. setupJob runs getFileStatus on the output path first and fails fast for every conflict-resolution mode (fail, append, replace) when the existing entry is a file, because a file at the output root cannot receive task output. The message text bundles the E_DEST_EXISTS constant, which makes it read like a conflict-mode complaint even though the deciding condition here is '!status.isDirectory()'.","triggerScenarios":"Starting a job whose mapreduce output path resolves to an existing S3 object (file), e.g. s3a://bucket/output where 'output' is a zero-byte marker object or a previously written single file.","commonSituations":"A prior run or manual upload left a file at the exact output path; console tools or other applications create an empty object as a path marker; a prior job with a single output file wrote directly to the output root.","solutions":["Delete the existing file at the destination path (hadoop fs -rm s3a://bucket/output) or choose a fresh output path","If the destination is an existing directory and you still want strict semantics, review fs.s3a.committer.staging.conflict-mode (values: fail, append, replace; default append) -- but note no mode tolerates a file at the destination","Add a job-level preflight check that lists the output path and fails with a clear message before the cluster is allocated"],"exampleFix":"# before: 'output' exists as a file from a prior run\nhadoop jar job.jar -Dmapreduce.output.fileoutputformat.outputdir=s3a://bucket/output ...\n\n# after: remove the stale object first\nhadoop fs -rm s3a://bucket/output\nhadoop jar job.jar -Dmapreduce.output.fileoutputformat.outputdir=s3a://bucket/output ...","handlingStrategy":"validation","validationCode":"try {\n  FileStatus st = fs.getFileStatus(outputPath);\n  if (!st.isDirectory()) {\n    throw new IOException(\"Output path exists as a FILE; remove it or choose a new path: \"\n        + outputPath);\n  }\n} catch (FileNotFoundException ok) {\n  // absent is fine\n}","typeGuard":null,"tryCatchPattern":"try {\n  committer.setupJob(context);\n} catch (PathExistsException e) {\n  // destination exists as a file: no conflict mode rescues this; remove/rename it and resubmit\n  LOG.error(\"Output destination occupied by a file: {}\", outputPath, e);\n}","preventionTips":["Run a preflight 'does the output path exist and is it a file' check in the job driver","Use unique per-run output paths (timestamps/run ids) in shared buckets","Do not create marker objects at intended output-directory locations"],"tags":["s3a","committer","conflict-resolution","output-path"],"backgroundTag":"output-path-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}