apache/hadoop · warning · IOException

already exists: {}

Error message

already exists: {}

What it means

SequenceFile.Sorter.sort(Path[], Path, boolean) refuses to overwrite: before any sorting it checks fs.exists(outFile) and throws IOException("already exists: <outFile>") if the destination is present. This is a safety guard against silently clobbering previous sort output, not a corruption signal.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:3012

    /**
     * Set the progressable object in order to report progress.
     * @param progressable input Progressable.
     */
    public void setProgressable(Progressable progressable) {
      this.progressable = progressable;
    }
    
    /** 
     * Perform a file sort from a set of input files into an output file.
     * @param inFiles the files to be sorted
     * @param outFile the sorted output file
     * @param deleteInput should the input files be deleted as they are read?
     * @throws IOException raised on errors performing I/O.
     */
    public void sort(Path[] inFiles, Path outFile,
                     boolean deleteInput) throws IOException {
      if (fs.exists(outFile)) {
        throw new IOException("already exists: " + outFile);
      }

      this.inFiles = inFiles;
      this.outFile = outFile;

      int segments = sortPass(deleteInput);
      if (segments > 1) {
        mergePass(outFile.getParent());
      }
    }

    /** 
     * Perform a file sort from a set of input files and return an iterator.
     * @param inFiles the files to be sorted
     * @param tempDir the directory where temp files are created during sort
     * @param deleteInput should the input files be deleted as they are read?
     * @return iterator the RawKeyValueIterator
     * @throws IOException raised on errors performing I/O.

View on GitHub (pinned to 2add963021)

Solutions

1. Delete or move the existing output before sorting: if (fs.exists(outFile)) fs.delete(outFile, true).
2. Write each run to a unique output path (append jobId/timestamp) and rename atomically on success.
3. Add cleanup of the output location in a finally block when runs fail.

Example fix

// before
sorter.sort(inFiles, outFile, false);    // throws if outFile exists

// after
if (fs.exists(outFile) && !fs.delete(outFile, true)) {
  throw new IOException("Cannot replace " + outFile);
}
sorter.sort(inFiles, outFile, false);

if (fs.exists(outFile)) {
  if (!fs.delete(outFile, true)) {
    throw new IOException("Cannot replace existing output " + outFile);
  }
}
sorter.sort(inFiles, outFile, deleteInput);

When it happens

Trigger: Calling sort() twice with the same output path in one run; rerunning an application that reuses a fixed output location; a previous failed run that wrote the output but never completed cleanup.

Common situations: Retried jobs or scripts with deterministic output names; shared working directories on HDFS or local FS; leftovers from crashed runs that were never cleaned.

Related errors

• File already exists: ${path}. Append or overwrite option mus
• mkdir of ${f} failed
• rename destination ${dst} already exists.
• {} already exists
• Cannot overwrite an existing file: %s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/25f767ae68bf3af5. Report an issue: GitHub.