apache/hadoop · error · IOException

Failed to remove %s: %s

Error message

Failed to remove %s: %s

What it means

When a storage directory fails to load, DataNode drops it from its list and calls sd.unlock() to release the storage lock. If unlock itself throws, each per-directory failure is formatted ('Failed to remove <root>: <reason>') into errorMsgBuilder; after the loop the accumulated text is thrown as one IOException summarizing every directory that could not be cleanly released.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:545

          BlockPoolSliceStorage bpsStorage = entry.getValue();
          File bpRoot =
              BlockPoolSliceStorage.getBpRoot(bpid, sd.getCurrentDir());
          bpsStorage.remove(bpRoot.getAbsoluteFile());
        }

        getStorageDirs().remove(sd);
        try {
          sd.unlock();
        } catch (IOException e) {
          LOG.warn("I/O error attempting to unlock storage directory {}.",
              sd.getRoot(), e);
          errorMsgBuilder.append(String.format("Failed to remove %s: %s%n",
              sd.getRoot(), e.getMessage()));
        }
      }
    }
    if (errorMsgBuilder.length() > 0) {
      throw new IOException(errorMsgBuilder.toString());
    }
  }

  /**
   * Analyze storage directories for a specific block pool.
   * Recover from previous transitions if required.
   * Perform fs state transition if necessary depending on the namespace info.
   * Read storage info.
   * <br>
   * This method should be synchronized between multiple DN threads.  Only the
   * first DN thread does DN level storage dir recoverTransitionRead.
   *
   * @param datanode DataNode
   * @param nsInfo Namespace info of namenode corresponding to the block pool
   * @param dataDirs Storage directories
   * @param startOpt startup option
   * @throws IOException on error
   */

View on GitHub (pinned to 2add963021)

Solutions

1. Find and stop the other process holding the lock: lsof / fuser on <dataDir>/in_use.lock
2. Check for a stale DN process on the host (jps, ps -ef | grep DataNode) and kill it
3. Fix filesystem/mount health - NFS soft mounts are unsupported for DN storage - and restart the DN
4. As a last resort, with the DN fully stopped, remove the stale lock file manually

# pre-start check: no other process should hold the storage locks
for d in /data*/hdfs/dfs/data; do
  fuser -v "$d/in_use.lock" && echo "LOCK HELD on $d" || true
done

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to remove")) {
    // one or more failed dirs also failed to unlock: find lock holders (lsof),
    // stop the stale DN process, then restart
  } else { throw e; }
}

When it happens

Trigger: A storage directory fails initial load (I/O error, bad state) during recoverTransitionRead/addStorageLocations and, additionally, its in_use.lock cannot be released - another process holds the lock stream, or the filesystem errors on close/delete.

Common situations: Two datanode processes (a stuck old DN plus a restart) pointed at the same dir; NFS or flaky mounts where lock release fails; device errors during startup.

Related errors

• All specified directories have failed to load.
• Root {}: DatanodeUuid={}, does not match {} from other Stora
• Cannot lock storage {root}. The directory is already locked
• Incompatible node types: storageType={storageType} but Stora
• Cluster IDs not matched: dn cid={clusterId} but ns cid={nsCi

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e7695f7799038bb8. Report an issue: GitHub.