apache/hadoop · error · DiskErrorException

File deletion failed!

Error message

File deletion failed!

What it means

The finally block of ReadWriteDiskValidator.checkStatus deletes the temp 'test*.tmp' file it created; if Files.delete throws IOException it also flags the failure metric and throws DiskErrorException("File deletion failed!", e). The write/read probe itself may already have passed — the volume is penalized purely because cleanup failed, so this error isolates a deletion-specific fault.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReadWriteDiskValidator.java:90

          System.nanoTime() - startTime, TimeUnit.NANOSECONDS);
      metric.addReadFileLatency(readLatency);

      // validation
      if (!Arrays.equals(inputBytes, outputBytes)) {
        metric.diskCheckFailed();
        throw new DiskErrorException("Data in file has been corrupted.");
      }
    } catch (IOException e) {
      metric.diskCheckFailed();
      throw new DiskErrorException("Disk Check failed!", e);
    } finally {
      // delete the file
      if (tmpFile != null) {
        try {
          Files.delete(tmpFile);
        } catch (IOException e) {
          metric.diskCheckFailed();
          throw new DiskErrorException("File deletion failed!", e);
        }
      }
    }
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Remove leftover test*.tmp files; lsattr them and chattr -i if immutable
2. Check /proc/mounts for an ro remount and repair the underlying disk/filesystem (dmesg shows the trigger)
3. Ensure no antivirus/backup agent locks files inside the checked directory
4. Restart the NodeManager and confirm DirectoryCollection re-marks the dir healthy

Example fix

# before
lsattr /mnt/yarn/local/test*.tmp   # shows 'i' flag

# after
sudo chattr -i /mnt/yarn/local/test*.tmp && sudo rm -f /mnt/yarn/local/test*.tmp

try { diskValidator.checkStatus(dir); } catch (DiskErrorException e) { if (e.getMessage() != null && e.getMessage().contains("File deletion failed")) { scheduleCleanup(dir, "test*.tmp"); } volume.markFailed(); }

When it happens

Trigger: Filesystem switched to read-only between create and delete (auto-remount after an I/O error); immutable attribute (chattr +i) on the file; external processes holding locks on the file on Windows-style mounts; accumulated stale test*.tmp files.

Common situations: Volumes that flip ro mid-check on flaky disks; security tooling making files immutable; shared mounts where another agent recreates or locks temp files.

Related errors

• dir + " is not a directory!"
• Data in file has been corrupted.
• Disk Check failed!
• Failed to delete temporary files while closing stream: '%s'
• failed to create tmp file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/9d12da356f2535bf. Report an issue: GitHub.