What causes "Failed to delete {dir}" in apache/hadoop?

Find open handles under the directory (lsof +D ) and close/stop the holders, then retry

apache/hadoop · error · IOException

Failed to delete {dir}

Error message

Failed to delete {dir}

What it means

Storage.deleteDir wraps FileUtil.fullyDelete, which recursively removes a directory's contents and then the directory itself, returning false on any failure. This IOException therefore means the recursive delete did not fully succeed - some entry could not be removed.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java:1388

          + "' to '" + destFile + "'");
    }
    if (preserveFileDate) {
      if (destFile.setLastModified(srcFile.lastModified()) == false) {
        LOG.debug("Failed to preserve last modified date from'{}' to '{}'",
            srcFile, destFile);
      }
    }
  }

  /**
   * Recursively delete all the content of the directory first and then 
   * the directory itself from the local filesystem.
   * @param dir The directory to delete
   * @throws IOException
   */
  public static void deleteDir(File dir) throws IOException {
    if (!FileUtil.fullyDelete(dir))
      throw new IOException("Failed to delete " + dir.getCanonicalPath());
  }
  
  /**
   * Write all data storage files.
   * @throws IOException
   */
  public void writeAll() throws IOException {
    this.layoutVersion = getServiceLayoutVersion();
    for (Iterator<StorageDirectory> it = storageDirs.iterator(); it.hasNext();) {
      writeProperties(it.next());
    }
  }

  /**
   * Unlock all storage directories.
   * @throws IOException
   */
  public void unlockAll() throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

Find open handles under the directory (lsof +D <dir>) and close/stop the holders, then retry
Run the cleanup as the directory's owning user, or fix ownership: chown -R hdfs:hdfs <dir>
Confirm the filesystem is mounted read-write and no SELinux denial appears in the audit log
Retry after the concurrent writer/reader has exited

Defensive patterns

Strategy: try-catch

Validate before calling

if (dir.exists() && (!dir.canRead() || !dir.canWrite())) {
  throw new IOException("cannot clean " + dir + " as current user; fix ownership first");
}

Try / catch

try {
  Storage.deleteDir(dir);
} catch (IOException e) {
  // list remaining entries, lsof them, stop holders, then retry once; otherwise rethrow
  throw e;
}

Prevention

Stop processes that hold files under the directory before cleanup
Run cleanup as the directory's owning user
Treat persistent fullyDelete failures as a signal of handles or SELinux, not of random corruption

When it happens

Trigger: A file inside the directory is held open by another process (Windows locking, NFS silly-rename); a subdirectory or file is not writable/executable by the daemon user; the volume is read-only; SELinux denies unlink; path components changed during deletion.

Common situations: Cleaning up old checkpoint or storage dirs while a daemon still holds files; cleanup scripts running as the wrong user; NFS-backed temp dirs; containers with leftover handles.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/586911034457085c. Report an issue: GitHub.