apache/hadoop · error · FileAlreadyExistsException

File already exists: ${path}. Append or overwrite option mus

Error message

File already exists: ${path}. Append or overwrite option must be specified in ${flag}

What it means

CreateFlag.validate(path, pathExists=true, flag) throws FileAlreadyExistsException when the target exists but the flags contain neither APPEND nor OVERWRITE. This is the guard against silently clobbering existing data: a plain CREATE against an existing file is treated as a user error rather than an implicit overwrite.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CreateFlag.java:179

    }
  }
  
  /**
   * Validate the CreateFlag for create operation
   * @param path Object representing the path; usually String or {@link Path}
   * @param pathExists pass true if the path exists in the file system
   * @param flag set of CreateFlag
   * @throws IOException on error
   * @throws HadoopIllegalArgumentException if the CreateFlag is invalid
   */
  public static void validate(Object path, boolean pathExists,
      EnumSet<CreateFlag> flag) throws IOException {
    validate(flag);
    final boolean append = flag.contains(APPEND);
    final boolean overwrite = flag.contains(OVERWRITE);
    if (pathExists) {
      if (!(append || overwrite)) {
        throw new FileAlreadyExistsException("File already exists: "
            + path.toString()
            + ". Append or overwrite option must be specified in " + flag);
      }
    } else if (!flag.contains(CREATE)) {
      throw new FileNotFoundException("Non existing file: " + path.toString()
          + ". Create option is not specified in " + flag);
    }
  }

  /**
   * Validate the CreateFlag for the append operation. The flag must contain
   * APPEND, and cannot contain OVERWRITE.
   *
   * @param flag enum set flag.
   */
  public static void validateForAppend(EnumSet<CreateFlag> flag) {
    validate(flag);
    if (!flag.contains(APPEND)) {

View on GitHub (pinned to 2add963021)

Solutions

1. If overwrite is intended, pass it: EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE) or fs.create(f, true) / builder.overwrite().
2. If append is intended (HDFS/raw-local), pass EnumSet.of(CreateFlag.APPEND) - though not for ChecksumFileSystem, which rejects append.
3. If the file is stale, delete it first (fs.delete(f, false)) and retry the create.
4. For job outputs, clean the output dir in setup (FileUtil.fullyDelete) or fail fast with a clear message naming the path.

Example fix

// before
FSDataOutputStream out = fs.create(path, false); // path exists -> FileAlreadyExistsException

// after (choose one intent)
FSDataOutputStream out = fs.create(path, true); // overwrite
// or: fs.delete(path, false); FSDataOutputStream out = fs.create(path, false);

boolean exists = fs.exists(path);
if (exists && !overwrite && !appendMode) {
  throw new FileAlreadyExistsException("refusing to clobber " + path);
}

try {
  out = fs.create(path, false);
} catch (FileAlreadyExistsException e) {
  // decide: overwrite (recreate with true), append, or delete-then-create
}

When it happens

Trigger: Calling fs.create(f, perms, EnumSet.of(CreateFlag.CREATE), ...) / fc.create equivalent / FSDataOutputStreamBuilder with create() but no overwrite() when f already exists; also the FileSystem.create(..., overwrite=false, ...) convenience forms which route through this validation.

Common situations: The classic MapReduce/Spark 'Output directory already exists' failure when re-running a job without deleting output; scheduled jobs whose previous run did not clean up; tools defaulting to no-clobber writing into a shared directory where the file name is taken.

Related errors

• ${flag} does not specify any options
• ${flag}Both append and overwrite options cannot be enabled.
• Non existing file: ${path}. Create option is not specified i
• ${flag} does not contain APPEND
• mkdir of ${f} failed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3ffca2f06204cbe5. Report an issue: GitHub.