{"record":{"id":"3ffca2f06204cbe5","repo":"apache/hadoop","slug":"file-already-exists-path-append-or-overwrite","errorCode":null,"errorMessage":"File already exists: ${path}. Append or overwrite option must be specified in ${flag}","messagePattern":"File already exists: (.+?)\\. Append or overwrite option must be specified in (.+?)","errorType":"exception","errorClass":"FileAlreadyExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CreateFlag.java","lineNumber":179,"sourceCode":"    }\n  }\n  \n  /**\n   * Validate the CreateFlag for create operation\n   * @param path Object representing the path; usually String or {@link Path}\n   * @param pathExists pass true if the path exists in the file system\n   * @param flag set of CreateFlag\n   * @throws IOException on error\n   * @throws HadoopIllegalArgumentException if the CreateFlag is invalid\n   */\n  public static void validate(Object path, boolean pathExists,\n      EnumSet<CreateFlag> flag) throws IOException {\n    validate(flag);\n    final boolean append = flag.contains(APPEND);\n    final boolean overwrite = flag.contains(OVERWRITE);\n    if (pathExists) {\n      if (!(append || overwrite)) {\n        throw new FileAlreadyExistsException(\"File already exists: \"\n            + path.toString()\n            + \". Append or overwrite option must be specified in \" + flag);\n      }\n    } else if (!flag.contains(CREATE)) {\n      throw new FileNotFoundException(\"Non existing file: \" + path.toString()\n          + \". Create option is not specified in \" + flag);\n    }\n  }\n\n  /**\n   * Validate the CreateFlag for the append operation. The flag must contain\n   * APPEND, and cannot contain OVERWRITE.\n   *\n   * @param flag enum set flag.\n   */\n  public static void validateForAppend(EnumSet<CreateFlag> flag) {\n    validate(flag);\n    if (!flag.contains(APPEND)) {","sourceCodeStart":161,"sourceCodeEnd":197,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CreateFlag.java#L161-L197","documentation":"CreateFlag.validate(path, pathExists=true, flag) throws FileAlreadyExistsException when the target exists but the flags contain neither APPEND nor OVERWRITE. This is the guard against silently clobbering existing data: a plain CREATE against an existing file is treated as a user error rather than an implicit overwrite.","triggerScenarios":"Calling fs.create(f, perms, EnumSet.of(CreateFlag.CREATE), ...) / fc.create equivalent / FSDataOutputStreamBuilder with create() but no overwrite() when f already exists; also the FileSystem.create(..., overwrite=false, ...) convenience forms which route through this validation.","commonSituations":"The classic MapReduce/Spark 'Output directory already exists' failure when re-running a job without deleting output; scheduled jobs whose previous run did not clean up; tools defaulting to no-clobber writing into a shared directory where the file name is taken.","solutions":["If overwrite is intended, pass it: EnumSet.of(CreateFlag.CREATE, CreateFlag.OVERWRITE) or fs.create(f, true) / builder.overwrite().","If append is intended (HDFS/raw-local), pass EnumSet.of(CreateFlag.APPEND) - though not for ChecksumFileSystem, which rejects append.","If the file is stale, delete it first (fs.delete(f, false)) and retry the create.","For job outputs, clean the output dir in setup (FileUtil.fullyDelete) or fail fast with a clear message naming the path."],"exampleFix":"// before\nFSDataOutputStream out = fs.create(path, false); // path exists -> FileAlreadyExistsException\n\n// after (choose one intent)\nFSDataOutputStream out = fs.create(path, true); // overwrite\n// or: fs.delete(path, false); FSDataOutputStream out = fs.create(path, false);","handlingStrategy":"validation","validationCode":"boolean exists = fs.exists(path);\nif (exists && !overwrite && !appendMode) {\n  throw new FileAlreadyExistsException(\"refusing to clobber \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n  out = fs.create(path, false);\n} catch (FileAlreadyExistsException e) {\n  // decide: overwrite (recreate with true), append, or delete-then-create\n}","preventionTips":["Delete or move job outputs before re-running jobs; make cleanup part of the run script.","Pass overwrite=true only after confirming clobbering is safe.","Fail fast in setup with a clear message instead of deep in the first create."],"tags":["hadoop","filesystem","file-exists","create-flag","overwrite","job-output"],"backgroundTag":"file-already-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}