{"record":{"id":"cd46ecc8cafbbb0a","repo":"apache/hadoop","slug":"failed-to-delete-count-out-of-total-replica","errorCode":null,"errorMessage":"Failed to delete {count} (out of {total}) replica(s): {errorList}","messagePattern":"Failed to delete (.+?) \\(out of (.+?)\\) replica\\(s\\): (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":2423,"sourceCode":"              dataStorage.getTrashDirectoryForReplica(bpid, info));\n        } else {\n          asyncDiskService.deleteSync(v.obtainReference(), info,\n              new ExtendedBlock(bpid, invalidBlks[i]),\n              dataStorage.getTrashDirectoryForReplica(bpid, info));\n        }\n      } catch (ClosedChannelException e) {\n        LOG.warn(\"Volume {} is closed, ignore the deletion task for \" +\n            \"block: {}\", v, invalidBlks[i]);\n      }\n    }\n    if (!errors.isEmpty()) {\n      StringBuilder b = new StringBuilder(\"Failed to delete \")\n        .append(errors.size()).append(\" (out of \").append(invalidBlks.length)\n        .append(\") replica(s):\");\n      for(int i = 0; i < errors.size(); i++) {\n        b.append(\"\\n\").append(i).append(\") \").append(errors.get(i));\n      }\n      throw new IOException(b.toString());\n    }\n  }\n\n  /**\n   * Invalidate a block but does not delete the actual on-disk block file.\n   *\n   * It should only be used when deactivating disks.\n   *\n   * @param bpid the block pool ID.\n   * @param block The block to be invalidated.\n   */\n  public void invalidate(String bpid, ReplicaInfo block) {\n    // If a DFSClient has the replica in its cache of short-circuit file\n    // descriptors (and the client is using ShortCircuitShm), invalidate it.\n    datanode.getShortCircuitRegistry().processBlockInvalidation(\n        new ExtendedBlockId(block.getBlockId(), bpid));\n\n    // If the block is cached, start uncaching it.","sourceCodeStart":2405,"sourceCodeEnd":2441,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L2405-L2441","documentation":"IOException thrown by FsDatasetImpl.invalidate (block deletion path) summarizing per-block failures: it collects errors for each block that could not be deleted and reports 'Failed to delete N (out of M) replica(s):' followed by an indexed error list. Closed channels are skipped with only a warn; everything else (missing file, permission, disk error) lands in this aggregate. The NameNode instructs invalidation, so failure means unwanted replicas persist.","triggerScenarios":"invalidate(bpid, invalidBlks[], async) processing a NameNode BLOCK_INVALIDATE request where one or more delete() calls fail - e.g., permission denied on the block file, I/O error from a failing disk, or path already gone in a racy way that still errored.","commonSituations":"Data directory permissions changed (chown/chmod by an admin); disk developing bad sectors; dfs.datanode.data.dir on NFS with stale handles; SELinux/AppArmor denying unlink.","solutions":["Read the per-index error list in the message - it names the exact per-block cause (Permission denied, I/O error, ...).","Fix filesystem permissions/ownership on data.dir so the DataNode user can delete files.","Check disk health (dmesg, smartctl) and replace failing volumes.","Restart the DataNode after fixing the cause; repeated invalidation from the NameNode will then succeed and clear the replicas."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"boolean isPartialInvalidateFailure(IOException e) {\n  return e.getMessage() != null && e.getMessage().startsWith(\"Failed to delete \")\n      && e.getMessage().contains(\"replica(s)\");\n}","tryCatchPattern":"try {\n  dataset.invalidate(bpid, blocks);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Failed to delete\")) {\n    // per-block causes are listed in the message; fix fs-level cause, then the NN re-issues invalidation\n    logPerBlockCauses(e);\n    scheduleRetry(); // NameNode will resend the delete list\n  }\n}","preventionTips":["Keep data.dir owned and writable by the DataNode user; audit chmod/chown changes.","Monitor DataNode disk errors and replace failing volumes promptly.","Avoid NFS for data dirs; stale file handles surface as delete failures.","Check SELinux/AppArmor policies allow unlink in data directories."],"tags":["hdfs","datanode","invalidate","delete-failed","permissions","disk-failure"],"backgroundTag":"block-delete-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}