{"record":{"id":"e31c69da18120c33","repo":"apache/hadoop","slug":"failed-to-delete-file","errorCode":null,"errorMessage":"Failed to delete ${file}","messagePattern":"Failed to delete (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java","lineNumber":295,"sourceCode":"   * Try to perform some disk IO by writing to the given file\n   * without using Native IO.\n   *\n   * @param file\n   * @throws IOException if there was a non-retriable error.\n   */\n  private static void diskIoCheckWithoutNativeIo(File file)\n      throws IOException {\n    FileOutputStream fos = null;\n\n    try {\n      final FileIoProvider provider = fileIoProvider.get();\n      fos = provider.get(file);\n      provider.write(fos, new byte[1]);\n      fos.getFD().sync();\n      fos.close();\n      fos = null;\n      if (!file.delete() && file.exists()) {\n        throw new IOException(\"Failed to delete \" + file);\n      }\n      file = null;\n    } finally {\n      IOUtils.cleanupWithLogger(LOG, fos);\n      FileUtils.deleteQuietly(file);\n    }\n  }\n\n  /**\n   * Generate a path name for a test file under the given directory.\n   *\n   * @return file object.\n   */\n  @VisibleForTesting\n  static File getFileNameForDiskIoCheck(File dir, int iterationCount) {\n    if (iterationCount < DISK_IO_MAX_ITERATIONS) {\n      // Use file names of the format prefix.001 by default.\n      return new File(dir,","sourceCodeStart":277,"sourceCodeEnd":313,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java#L277-L313","documentation":"Inside diskIoCheckWithoutNativeIo the 1-byte probe write and fsync succeeded, but File.delete() returned false while the file still exists, raising IOException('Failed to delete <probe file>'). After the retry loop it surfaces wrapped inside 'Error checking directory' (the sibling error), so the directory is declared bad even though writes work. The probe name is generated by getFileNameForDiskIoCheck(dir, iteration).","triggerScenarios":"Filesystems where unlink behaves oddly: NFS silly-rename of recently-closed files (.nfsXXXX); overlayfs/FUSE quirks; antivirus or LSM hooks intercepting unlink; another process holding the file; metadata corruption on a failing disk.","commonSituations":"Hadoop local dirs placed on NFS mounts; containerized hosts with restrictive overlay filesystems; agents/AV scanners touching probe files; disks near failure whose metadata operations start failing.","solutions":["Keep Hadoop local dirs on local POSIX storage, not NFS","lsof the probe file to find external holders; stop or exclude the interferer (AV/agent)","Delete stray probe files manually and fsck the filesystem","If deletes keep failing, treat the disk as bad and fence it as the health check already does"],"exampleFix":"# before: local dirs on NFS, delete() silly-renames the probe file\nyarn.nodemanager.local-dirs = /mnt/nfs-share/nm-local\n# after: local storage\nyarn.nodemanager.local-dirs = /data/nm-local","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  DiskChecker.checkDirWithDiskIo(dir);\n} catch (DiskErrorException e) {\n  Throwable c = e.getCause();\n  if (c instanceof IOException && c.getMessage() != null && c.getMessage().startsWith(\"Failed to delete\")) {\n    // unlink problem (NFS silly-rename, holder process, corrupt FS): investigate, then fence\n    log.warn(\"delete of probe file failed on {}\", dir);\n  }\n  markDirFailed(dir);\n}","preventionTips":["Keep local dirs off NFS and exotic FUSE mounts","lsof probe files that fail to delete to find AV/agent interferers","Clean stray probe files and fsck the volume when deletes start failing"],"tags":["disk","filesystem","nfs","delete","diskchecker","health-check"],"backgroundTag":"file-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}