apache/hadoop · error · IOException

Failed to delete ${file}

Error message

Failed to delete ${file}

What it means

Inside diskIoCheckWithoutNativeIo the 1-byte probe write and fsync succeeded, but File.delete() returned false while the file still exists, raising IOException('Failed to delete <probe file>'). After the retry loop it surfaces wrapped inside 'Error checking directory' (the sibling error), so the directory is declared bad even though writes work. The probe name is generated by getFileNameForDiskIoCheck(dir, iteration).

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/DiskChecker.java:295

   * Try to perform some disk IO by writing to the given file
   * without using Native IO.
   *
   * @param file
   * @throws IOException if there was a non-retriable error.
   */
  private static void diskIoCheckWithoutNativeIo(File file)
      throws IOException {
    FileOutputStream fos = null;

    try {
      final FileIoProvider provider = fileIoProvider.get();
      fos = provider.get(file);
      provider.write(fos, new byte[1]);
      fos.getFD().sync();
      fos.close();
      fos = null;
      if (!file.delete() && file.exists()) {
        throw new IOException("Failed to delete " + file);
      }
      file = null;
    } finally {
      IOUtils.cleanupWithLogger(LOG, fos);
      FileUtils.deleteQuietly(file);
    }
  }

  /**
   * Generate a path name for a test file under the given directory.
   *
   * @return file object.
   */
  @VisibleForTesting
  static File getFileNameForDiskIoCheck(File dir, int iterationCount) {
    if (iterationCount < DISK_IO_MAX_ITERATIONS) {
      // Use file names of the format prefix.001 by default.
      return new File(dir,

View on GitHub (pinned to 2add963021)

Solutions

1. Keep Hadoop local dirs on local POSIX storage, not NFS
2. lsof the probe file to find external holders; stop or exclude the interferer (AV/agent)
3. Delete stray probe files manually and fsck the filesystem
4. If deletes keep failing, treat the disk as bad and fence it as the health check already does

Example fix

# before: local dirs on NFS, delete() silly-renames the probe file
yarn.nodemanager.local-dirs = /mnt/nfs-share/nm-local
# after: local storage
yarn.nodemanager.local-dirs = /data/nm-local

try {
  DiskChecker.checkDirWithDiskIo(dir);
} catch (DiskErrorException e) {
  Throwable c = e.getCause();
  if (c instanceof IOException && c.getMessage() != null && c.getMessage().startsWith("Failed to delete")) {
    // unlink problem (NFS silly-rename, holder process, corrupt FS): investigate, then fence
    log.warn("delete of probe file failed on {}", dir);
  }
  markDirFailed(dir);
}

When it happens

Trigger: Filesystems where unlink behaves oddly: NFS silly-rename of recently-closed files (.nfsXXXX); overlayfs/FUSE quirks; antivirus or LSM hooks intercepting unlink; another process holding the file; metadata corruption on a failing disk.

Common situations: Hadoop local dirs placed on NFS mounts; containerized hosts with restrictive overlay filesystems; agents/AV scanners touching probe files; disks near failure whose metadata operations start failing.

Related errors

• Cannot create directory: ${dir}
• Not a directory: ${dir}
• Directory is not readable: ${dir}
• Directory is not writable: ${dir}
• Directory is not executable: ${dir}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e31c69da18120c33. Report an issue: GitHub.