{"record":{"id":"6f59b25f946e35c5","repo":"apache/hadoop","slug":"output-directory-already-exists-6f59b2","errorCode":null,"errorMessage":"Output directory {} already exists","messagePattern":"Output directory (.+?) already exists","errorType":"validation","errorClass":"FileAlreadyExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java","lineNumber":164,"sourceCode":"  \n  public abstract RecordWriter<K, V> \n     getRecordWriter(TaskAttemptContext job\n                     ) throws IOException, InterruptedException;\n\n  public void checkOutputSpecs(JobContext job\n                               ) throws FileAlreadyExistsException, IOException{\n    // Ensure that the output directory is set and not already there\n    Path outDir = getOutputPath(job);\n    if (outDir == null) {\n      throw new InvalidJobConfException(\"Output directory not set.\");\n    }\n\n    // get delegation token for outDir's file system\n    TokenCache.obtainTokensForNamenodes(job.getCredentials(),\n        new Path[] { outDir }, job.getConfiguration());\n\n    if (outDir.getFileSystem(job.getConfiguration()).exists(outDir)) {\n      throw new FileAlreadyExistsException(\"Output directory \" + outDir + \n                                           \" already exists\");\n    }\n  }\n\n  /**\n   * Set the {@link Path} of the output directory for the map-reduce job.\n   *\n   * @param job The job to modify\n   * @param outputDir the {@link Path} of the output directory for \n   * the map-reduce job.\n   */\n  public static void setOutputPath(Job job, Path outputDir) {\n    try {\n      outputDir = outputDir.getFileSystem(job.getConfiguration()).makeQualified(\n          outputDir);\n    } catch (IOException e) {\n        // Throw the IOException as a RuntimeException to be compatible with MR1\n        throw new RuntimeException(e);","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java#L146-L182","documentation":"Thrown as FileAlreadyExistsException from FileOutputFormat.checkOutputSpecs (FileOutputFormat.java:164) when the configured output directory already exists on its FileSystem at job submission time. This is a deliberate fail-fast guard: MapReduce output directories must be created fresh per job, because the committer assumes exclusive ownership of <outdir>/_temporary.","triggerScenarios":"checkOutputSpecs() at job submit finds outDir.getFileSystem(conf).exists(outDir) == true. Concrete triggers: re-running a failed/successful job with the same hardcoded output path, a scheduler/Oozie workflow retrying into the same directory, or two jobs submitted with identical output paths.","commonSituations":"The single most common MR submission error: development iterations that rerun the same command; automated retries; scheduled jobs that write to a path created by the previous run.","solutions":["Delete or move the existing directory: hadoop fs -rm -r <outdir> (only if its contents are disposable)","Give each run a unique output path, e.g. append a timestamp or run id: /data/out-20260822T1015","In Oozie/Airflow-style pipelines, add a pre-step that archives (<outdir> -> <outdir>.bkup-<ts>) or removes the path before submission","If the output should be additive instead, use HDFS append-based tools or write to a new partition directory rather than the job root"],"exampleFix":"// before\nFileOutputFormat.setOutputPath(job, new Path(\"/data/out\")); // exists -> submit fails\n\n// after: unique path per run\nFileOutputFormat.setOutputPath(job,\n    new Path(\"/data/out-\" + java.time.LocalDate.now()));","handlingStrategy":"validation","validationCode":"// before submit: replicate checkOutputSpecs' guard early with a clear message\nPath out = FileOutputFormat.getOutputPath(job);\nFileSystem fs = out.getFileSystem(conf);\nif (fs.exists(out)) {\n  throw new IllegalStateException(\"Output dir already exists: \" + out\n      + \" — delete it or use a per-run path\");\n}","typeGuard":null,"tryCatchPattern":"catch FileAlreadyExistsException around job.submit()/waitForCompletion(); on catch, archive the old dir (fs.rename(out, archivePath)) and resubmit once — do not blind-retry","preventionTips":["Derive output paths from run id/timestamp/partition date","In scheduled pipelines, add a cleanup/archive pre-step","Never point two concurrent jobs at the same output dir"],"tags":["hadoop","mapreduce","output-path","file-system","job-submission","filealreadyexists"],"backgroundTag":"output-directory-already-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}