apache/hadoop · error · IOException

Unable to delete {}

Error message

Unable to delete {}

What it means

During recovery of a crashed pre-transactional checkpoint: if both fsimage.ckpt and edits.new exist, the inspector cannot know whether the SecondaryNameNode finished uploading the merged image, so it discards the checkpoint file. File.delete() returning false is turned into this IOException — an environment-level failure: missing permission, a file held open/locked, or a read-only filesystem.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImagePreTransactionalStorageInspector.java:222

      NNStorage.getStorageFile(latestNameSD, NameNodeFile.IMAGE);
    File ckptFile =
      NNStorage.getStorageFile(latestNameSD, NameNodeFile.IMAGE_NEW);
    
    //
    // If we were in the midst of a checkpoint
    //
    if (ckptFile.exists()) {
      needToSave = true;
      if (NNStorage.getStorageFile(latestEditsSD, NameNodeFile.EDITS_NEW)
          .exists()) {
        //
        // checkpointing migth have uploaded a new
        // merged image, but we discard it here because we are
        // not sure whether the entire merged image was uploaded
        // before the namenode crashed.
        //
        if (!ckptFile.delete()) {
          throw new IOException("Unable to delete " + ckptFile);
        }
      } else {
        //
        // checkpointing was in progress when the namenode
        // shutdown. The fsimage.ckpt was created and the edits.new
        // file was moved to edits. We complete that checkpoint by
        // moving fsimage.new to fsimage. There is no need to 
        // update the fstime file here. renameTo fails on Windows
        // if the destination file already exists.
        //
        if (!ckptFile.renameTo(curFile)) {
          if (!curFile.delete())
            LOG.warn("Unable to delete dir " + curFile + " before rename");
          if (!ckptFile.renameTo(curFile)) {
            throw new IOException("Unable to rename " + ckptFile +
                                  " to " + curFile);
          }
        }

View on GitHub (pinned to 2add963021)

Solutions

1. Fix ownership/permissions so the NN user can write the name dir: chown -R hdfs:hdfs /dfs/name
2. Check for a stale in_use.lock or a second NameNode running against the same directories
3. Remount read-only filesystems read-write, or move storage off the broken NFS volume
4. Once you confirm no checkpoint is in flight, delete the stale fsimage.ckpt and edits.new manually and restart the NameNode

Example fix

# before: NN user cannot delete the checkpoint file
ls -l /dfs/name/current/fsimage.ckpt   # owned by root
# after
chown -R hdfs:hdfs /dfs/name && su - hdfs -c 'hdfs --daemon start namenode'

for (URI u : FSNamesystem.getNamespaceDirs(conf)) {
  File cur = new File(u.getPath(), "current");
  if (!cur.isDirectory() || !Files.isWritable(cur.toPath())) {
    throw new IOException("name dir not writable by NameNode user: " + cur);
  }
}

When it happens

Trigger: NameNode restart while fsimage.ckpt and edits.new both exist in current/, and java.io.File.delete() fails on ckptFile (permission denied, read-only mount, Windows file lock, stale NFS handle).

Common situations: name directory owned by root while the NN runs as hdfs; storage on a read-only or broken NFS export; antivirus or a lingering process holding the file open on Windows.

Related errors

• Unable to rename {} to {}
• Image file is not found in {}
• Edits file is not found in {}
• Inconsistent storage detected, image and edits checkpoint ti
• Unexpected HAServiceStateProto:

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f1c56533e07b6912. Report an issue: GitHub.