apache/hadoop · error · FileAlreadyExistsException

Output directory {} already exists

Error message

Output directory {} already exists

What it means

Thrown as FileAlreadyExistsException from FileOutputFormat.checkOutputSpecs (FileOutputFormat.java:164) when the configured output directory already exists on its FileSystem at job submission time. This is a deliberate fail-fast guard: MapReduce output directories must be created fresh per job, because the committer assumes exclusive ownership of <outdir>/_temporary.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java:164

  
  public abstract RecordWriter<K, V> 
     getRecordWriter(TaskAttemptContext job
                     ) throws IOException, InterruptedException;

  public void checkOutputSpecs(JobContext job
                               ) throws FileAlreadyExistsException, IOException{
    // Ensure that the output directory is set and not already there
    Path outDir = getOutputPath(job);
    if (outDir == null) {
      throw new InvalidJobConfException("Output directory not set.");
    }

    // get delegation token for outDir's file system
    TokenCache.obtainTokensForNamenodes(job.getCredentials(),
        new Path[] { outDir }, job.getConfiguration());

    if (outDir.getFileSystem(job.getConfiguration()).exists(outDir)) {
      throw new FileAlreadyExistsException("Output directory " + outDir + 
                                           " already exists");
    }
  }

  /**
   * Set the {@link Path} of the output directory for the map-reduce job.
   *
   * @param job The job to modify
   * @param outputDir the {@link Path} of the output directory for 
   * the map-reduce job.
   */
  public static void setOutputPath(Job job, Path outputDir) {
    try {
      outputDir = outputDir.getFileSystem(job.getConfiguration()).makeQualified(
          outputDir);
    } catch (IOException e) {
        // Throw the IOException as a RuntimeException to be compatible with MR1
        throw new RuntimeException(e);

View on GitHub (pinned to 2add963021)

Solutions

1. Delete or move the existing directory: hadoop fs -rm -r <outdir> (only if its contents are disposable)
2. Give each run a unique output path, e.g. append a timestamp or run id: /data/out-20260822T1015
3. In Oozie/Airflow-style pipelines, add a pre-step that archives (<outdir> -> <outdir>.bkup-<ts>) or removes the path before submission
4. If the output should be additive instead, use HDFS append-based tools or write to a new partition directory rather than the job root

Example fix

// before
FileOutputFormat.setOutputPath(job, new Path("/data/out")); // exists -> submit fails

// after: unique path per run
FileOutputFormat.setOutputPath(job,
    new Path("/data/out-" + java.time.LocalDate.now()));

// before submit: replicate checkOutputSpecs' guard early with a clear message
Path out = FileOutputFormat.getOutputPath(job);
FileSystem fs = out.getFileSystem(conf);
if (fs.exists(out)) {
  throw new IllegalStateException("Output dir already exists: " + out
      + " — delete it or use a per-run path");
}

catch FileAlreadyExistsException around job.submit()/waitForCompletion(); on catch, archive the old dir (fs.rename(out, archivePath)) and resubmit once — do not blind-retry

When it happens

Trigger: checkOutputSpecs() at job submit finds outDir.getFileSystem(conf).exists(outDir) == true. Concrete triggers: re-running a failed/successful job with the same hardcoded output path, a scheduler/Oozie workflow retrying into the same directory, or two jobs submitted with identical output paths.

Common situations: The single most common MR submission error: development iterations that rerun the same command; automated retries; scheduled jobs that write to a path created by the previous run.

Related errors

• Output directory not set.
• Invalid specification for distributed-cache artifacts of typ
• SequenceFileAsBinaryOutputFormat doesn't support Record Comp
• Failed to run job : {diagnostics}
• {} is a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6f59b25f946e35c5. Report an issue: GitHub.