{"record":{"id":"e7695f7799038bb8","repo":"apache/hadoop","slug":"failed-to-remove-s-s","errorCode":null,"errorMessage":"Failed to remove %s: %s","messagePattern":"Failed to remove (.+?): (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java","lineNumber":545,"sourceCode":"          BlockPoolSliceStorage bpsStorage = entry.getValue();\n          File bpRoot =\n              BlockPoolSliceStorage.getBpRoot(bpid, sd.getCurrentDir());\n          bpsStorage.remove(bpRoot.getAbsoluteFile());\n        }\n\n        getStorageDirs().remove(sd);\n        try {\n          sd.unlock();\n        } catch (IOException e) {\n          LOG.warn(\"I/O error attempting to unlock storage directory {}.\",\n              sd.getRoot(), e);\n          errorMsgBuilder.append(String.format(\"Failed to remove %s: %s%n\",\n              sd.getRoot(), e.getMessage()));\n        }\n      }\n    }\n    if (errorMsgBuilder.length() > 0) {\n      throw new IOException(errorMsgBuilder.toString());\n    }\n  }\n\n  /**\n   * Analyze storage directories for a specific block pool.\n   * Recover from previous transitions if required.\n   * Perform fs state transition if necessary depending on the namespace info.\n   * Read storage info.\n   * <br>\n   * This method should be synchronized between multiple DN threads.  Only the\n   * first DN thread does DN level storage dir recoverTransitionRead.\n   *\n   * @param datanode DataNode\n   * @param nsInfo Namespace info of namenode corresponding to the block pool\n   * @param dataDirs Storage directories\n   * @param startOpt startup option\n   * @throws IOException on error\n   */","sourceCodeStart":527,"sourceCodeEnd":563,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java#L527-L563","documentation":"When a storage directory fails to load, DataNode drops it from its list and calls sd.unlock() to release the storage lock. If unlock itself throws, each per-directory failure is formatted ('Failed to remove <root>: <reason>') into errorMsgBuilder; after the loop the accumulated text is thrown as one IOException summarizing every directory that could not be cleanly released.","triggerScenarios":"A storage directory fails initial load (I/O error, bad state) during recoverTransitionRead/addStorageLocations and, additionally, its in_use.lock cannot be released - another process holds the lock stream, or the filesystem errors on close/delete.","commonSituations":"Two datanode processes (a stuck old DN plus a restart) pointed at the same dir; NFS or flaky mounts where lock release fails; device errors during startup.","solutions":["Find and stop the other process holding the lock: lsof / fuser on <dataDir>/in_use.lock","Check for a stale DN process on the host (jps, ps -ef | grep DataNode) and kill it","Fix filesystem/mount health - NFS soft mounts are unsupported for DN storage - and restart the DN","As a last resort, with the DN fully stopped, remove the stale lock file manually"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"# pre-start check: no other process should hold the storage locks\nfor d in /data*/hdfs/dfs/data; do\n  fuser -v \"$d/in_use.lock\" && echo \"LOCK HELD on $d\" || true\ndone","typeGuard":null,"tryCatchPattern":"catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Failed to remove\")) {\n    // one or more failed dirs also failed to unlock: find lock holders (lsof),\n    // stop the stale DN process, then restart\n  } else { throw e; }\n}","preventionTips":["Ensure exactly one datanode process per storage directory set; use systemd/fencing to prevent doubles","Avoid NFS for DN storage where lock release semantics are unreliable","When a DN is killed -9, verify locks are released before restarting it"],"tags":["hadoop","hdfs","datanode","storage","file-lock","startup"],"backgroundTag":"file-lock-release-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}