apache/hadoop · error · IOException

Failed to remove

Error message

Failed to remove %s: %s

What it means

When a storage directory fails to load, DataNode drops it from its list and calls sd.unlock() to release the storage lock. If unlock itself throws, each per-directory failure is formatted ('Failed to remove <root>: <reason>') into errorMsgBuilder; after the loop the accumulated text is thrown as one IOException summarizing every directory that could not be cleanly released.

Solutions

1. Find and stop the other process holding the lock: lsof / fuser on <dataDir>/in_use.lock
2. Check for a stale DN process on the host (jps, ps -ef | grep DataNode) and kill it
3. Fix filesystem/mount health - NFS soft mounts are unsupported for DN storage - and restart the DN
4. As a last resort, with the DN fully stopped, remove the stale lock file manually

# pre-start check: no other process should hold the storage locks
for d in /data*/hdfs/dfs/data; do
  fuser -v "$d/in_use.lock" && echo "LOCK HELD on $d" || true
done

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Failed to remove")) {
    // one or more failed dirs also failed to unlock: find lock holders (lsof),
    // stop the stale DN process, then restart
  } else { throw e; }
}

When it happens

Trigger: A storage directory fails initial load (I/O error, bad state) during recoverTransitionRead/addStorageLocations and, additionally, its in_use.lock cannot be released - another process holds the lock stream, or the filesystem errors on close/delete.

Common situations: Two datanode processes (a stuck old DN plus a restart) pointed at the same dir; NFS or flaky mounts where lock release fails; device errors during startup.

Related errors

• All specified directories have failed to load.
• Root : DatanodeUuid= , does not match from other…
• Although a UNIX domain socket path is configured as
• BUG: The stored LV =
• Cannot lock storage . The directory is already locked

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e7695f7799038bb8. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:545

          BlockPoolSliceStorage bpsStorage = entry.getValue();
          File bpRoot =
              BlockPoolSliceStorage.getBpRoot(bpid, sd.getCurrentDir());
          bpsStorage.remove(bpRoot.getAbsoluteFile());
        }

        getStorageDirs().remove(sd);
        try {
          sd.unlock();
        } catch (IOException e) {
          LOG.warn("I/O error attempting to unlock storage directory {}.",
              sd.getRoot(), e);
          errorMsgBuilder.append(String.format("Failed to remove %s: %s%n",
              sd.getRoot(), e.getMessage()));
        }
      }
    }
    if (errorMsgBuilder.length() > 0) {
      throw new IOException(errorMsgBuilder.toString());
    }
  }

  /**
   * Analyze storage directories for a specific block pool.
   * Recover from previous transitions if required.
   * Perform fs state transition if necessary depending on the namespace info.
   * Read storage info.
   * <br>
   * This method should be synchronized between multiple DN threads.  Only the
   * first DN thread does DN level storage dir recoverTransitionRead.
   *
   * @param datanode DataNode
   * @param nsInfo Namespace info of namenode corresponding to the block pool
   * @param dataDirs Storage directories
   * @param startOpt startup option
   * @throws IOException on error
   */

View on GitHub (pinned to 2add963021)