{"record":{"id":"c4d80bf34e0fde7d","repo":"apache/hadoop","slug":"operation-path-path-already-exists-and-has-st","errorCode":null,"errorMessage":"{operation}: path {path} already exists and has status {status}","messagePattern":"(.+?): path (.+?) already exists and has status (.+?)","errorType":"exception","errorClass":"FileAlreadyExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/AbstractJobOrTaskStage.java","lineNumber":578,"sourceCode":"   * mkdirs() failed.\n   * @param operation operation for error reporting.\n   * @param path path path to create.\n   * @return the path.\n   * @throws IOException failure\n   * @throws PathIOException mkdirs failed.\n   * @throws FileAlreadyExistsException destination exists.\n   */\n  protected final Path createNewDirectory(\n      final String operation,\n      final Path path) throws IOException {\n    LOG.trace(\"{}: {} createNewDirectory('{}')\", getName(), operation, path);\n    requireNonNull(path,\n        () -> String.format(\"%s: Null path for operation %s\", getName(), operation));\n    // check for dir existence before trying to create.\n    try {\n      final FileStatus status = getFileStatus(path);\n      // no exception, so the path exists.\n      throw new FileAlreadyExistsException(operation\n          + \": path \" + path\n          + \" already exists and has status \" + status);\n    } catch (FileNotFoundException e) {\n      // the path does not exist, so create it.\n      mkdirs(path, true);\n      return path;\n    }\n  }\n\n  /**\n   * Assert that a path is a directory which must exist.\n   * @param operation operation for error reporting.\n   * @param path path path to create.\n   * @return the path\n   * @throws IOException failure\n   * @throws PathIOException mkdirs failed.\n   * @throws FileAlreadyExistsException destination exists.\n   */","sourceCodeStart":560,"sourceCodeEnd":596,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/committer/manifest/stages/AbstractJobOrTaskStage.java#L560-L596","documentation":"createNewDirectory() implements create-if-absent semantics: it probes getFileStatus first, and if the probe succeeds (no FileNotFoundException) the path already exists, so it throws FileAlreadyExistsException('<operation>: path <path> already exists and has status <status>'). Manifest-committer job/job-attempt directories are expected to be new; an existing one means stale state or a concurrent committer.","triggerScenarios":"A manifest committer stage creating a directory (job attempt dir etc.) that already exists: leftover from a prior failed attempt with the same job id, a concurrent job committing into the same output tree, or manual recreation of committer directories.","commonSituations":"Resubmitting a failed job with the same job id into the same output without cleanup; two applications sharing an output path; operators 'pre-creating' committer directory structures; repeated AM retries racing the commit.","solutions":["Clean stale committer state: remove the manifest committer's intermediate tree under the output path (or the whole output dir if it is disposable) and rerun.","Write to a fresh, job-unique output directory per run.","Ensure only one job at a time commits into a given output tree.","Read the status in the message - it tells you whether you are colliding with a file or an earlier attempt's directory."],"exampleFix":"// before: rerunning a failed job into the same output\nPath out = new Path(\"abfs://container/data/job-out\"); // contains stale attempt dirs\n\n// after: clean or version the output before resubmitting\nFileSystem fs = out.getFileSystem(conf);\nif (fs.exists(out)) { fs.delete(out, true); } // only if disposable\n// or: Path out = new Path(\"abfs://container/data/job-out-\" + runId);","handlingStrategy":"validation","validationCode":"// before resubmitting into a reused output path, clear stale committer state\nFileSystem fs = outputDir.getFileSystem(conf);\nif (fs.exists(outputDir)) {\n  // whole-tree delete if disposable; otherwise remove the committer's intermediate dirs only\n  fs.delete(outputDir, true);\n}","typeGuard":null,"tryCatchPattern":"try {\n  job.waitForCompletion(true);\n} catch (FileAlreadyExistsException e) {\n  // stale attempt dirs: clean the output tree (or use a new output dir) and resubmit\n}","preventionTips":["Use run-unique output directories instead of a fixed shared path.","Never run two committing jobs against the same output tree concurrently.","After a failed run, clean committer intermediate state before resubmitting with the same job id."],"tags":["hadoop","manifest-committer","file-already-exists","idempotency","job-retry"],"backgroundTag":"directory-already-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}