{"record":{"id":"20f5ea6f36e00712","repo":"apache/hadoop","slug":"file-f-already-exists","errorCode":null,"errorMessage":"File {f} already exists","messagePattern":"File (.+?) already exists","errorType":"exception","errorClass":"AlreadyExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java","lineNumber":248,"sourceCode":"    try {\n      Stat stat = NativeIO.POSIX.getFstat(fis.getFD());\n      checkStat(f, stat.getOwner(), stat.getGroup(), expectedOwner,\n          expectedGroup);\n      success = true;\n      return fis;\n    } finally {\n      if (!success) {\n        fis.close();\n      }\n    }\n  }\n\n  private static FileOutputStream insecureCreateForWrite(File f,\n      int permissions) throws IOException {\n    // If we can't do real security, do a racy exists check followed by an\n    // open and chmod\n    if (f.exists()) {\n      throw new AlreadyExistsException(\"File \" + f + \" already exists\");\n    }\n    FileOutputStream fos = new FileOutputStream(f);\n    boolean success = false;\n    try {\n      rawFilesystem.setPermission(new Path(f.getAbsolutePath()),\n        new FsPermission((short)permissions));\n      success = true;\n      return fos;\n    } finally {\n      if (!success) {\n        fos.close();\n      }\n    }\n  }\n\n  /**\n   * Open the specified File for write access, ensuring that it does not exist.\n   * @param f the file that we want to create","sourceCodeStart":230,"sourceCodeEnd":266,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java#L230-L266","documentation":"When secure O_EXCL creation is unavailable (security disabled or native code missing, i.e. skipSecurity==true), SecureIOUtils.createForWrite falls back to insecureCreateForWrite: a racy exists() check followed by open+chmod. If the path exists at check time it throws AlreadyExistsException (an IOException subclass defined inside SecureIOUtils). The check is best-effort only; a file created between check and open would still be clobbered, which is why this path exists solely for the insecure mode.","triggerScenarios":"SecureIOUtils.createForWrite(path, permissions) with security off/native libs absent while path already exists: leftover files from a previous task attempt in local dirs, two processes racing for the same filename, or re-running a job against uncleaned scratch space.","commonSituations":"Retried map/reduce tasks reusing mapreduce.cluster.local.dir; NodeManager local dirs not cleaned after an unclean kill; duplicated attempt ids; manual test runs that left files behind.","solutions":["Delete or archive the existing file, then retry the call","Write to a unique name per attempt (append attempt id or UUID) instead of a fixed path","Clean the node's local scratch dirs (nm-local-dirs, mapreduce local dirs) when re-running jobs","Run with security enabled plus native libraries so the secure O_EXCL path is used instead of this racy fallback"],"exampleFix":"// before\nFileOutputStream fos = SecureIOUtils.createForWrite(f, 0644); // AlreadyExistsException on stale file\n\n// after\ntry {\n  fos = SecureIOUtils.createForWrite(f, 0644);\n} catch (SecureIOUtils.AlreadyExistsException e) {\n  throw new IOException(\"Stale output \" + f + \" already present; clean local dirs\", e);\n}","handlingStrategy":"try-catch","validationCode":"if (f.exists()) {\n  // stale output: clean up or pick a unique name before calling createForWrite\n  throw new IOException(f + \" already exists; remove it or use a unique attempt path\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  FileOutputStream fos = SecureIOUtils.createForWrite(f, perms);\n} catch (SecureIOUtils.AlreadyExistsException e) {\n  // recover: delete stale file, or retry with a unique name\n  f.delete();\n  fos = SecureIOUtils.createForWrite(f, perms);\n}","preventionTips":["Always derive output filenames from attempt/task ids so reruns cannot collide","Clean local scratch dirs between job reruns","Remember the insecure path is racy: under disabled security, catching this exception is a best-effort guard, not an atomic create"],"tags":["hadoop","file-io","race-condition","security"],"backgroundTag":"file-already-exists","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}