apache/hadoop · error · IOException

Failed to delete {count} (out of {total}) replica(s): {error

Error message

Failed to delete {count} (out of {total}) replica(s): {errorList}

What it means

IOException thrown by FsDatasetImpl.invalidate (block deletion path) summarizing per-block failures: it collects errors for each block that could not be deleted and reports 'Failed to delete N (out of M) replica(s):' followed by an indexed error list. Closed channels are skipped with only a warn; everything else (missing file, permission, disk error) lands in this aggregate. The NameNode instructs invalidation, so failure means unwanted replicas persist.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2423

              dataStorage.getTrashDirectoryForReplica(bpid, info));
        } else {
          asyncDiskService.deleteSync(v.obtainReference(), info,
              new ExtendedBlock(bpid, invalidBlks[i]),
              dataStorage.getTrashDirectoryForReplica(bpid, info));
        }
      } catch (ClosedChannelException e) {
        LOG.warn("Volume {} is closed, ignore the deletion task for " +
            "block: {}", v, invalidBlks[i]);
      }
    }
    if (!errors.isEmpty()) {
      StringBuilder b = new StringBuilder("Failed to delete ")
        .append(errors.size()).append(" (out of ").append(invalidBlks.length)
        .append(") replica(s):");
      for(int i = 0; i < errors.size(); i++) {
        b.append("\n").append(i).append(") ").append(errors.get(i));
      }
      throw new IOException(b.toString());
    }
  }

  /**
   * Invalidate a block but does not delete the actual on-disk block file.
   *
   * It should only be used when deactivating disks.
   *
   * @param bpid the block pool ID.
   * @param block The block to be invalidated.
   */
  public void invalidate(String bpid, ReplicaInfo block) {
    // If a DFSClient has the replica in its cache of short-circuit file
    // descriptors (and the client is using ShortCircuitShm), invalidate it.
    datanode.getShortCircuitRegistry().processBlockInvalidation(
        new ExtendedBlockId(block.getBlockId(), bpid));

    // If the block is cached, start uncaching it.

View on GitHub (pinned to 2add963021)

Solutions

1. Read the per-index error list in the message - it names the exact per-block cause (Permission denied, I/O error, ...).
2. Fix filesystem permissions/ownership on data.dir so the DataNode user can delete files.
3. Check disk health (dmesg, smartctl) and replace failing volumes.
4. Restart the DataNode after fixing the cause; repeated invalidation from the NameNode will then succeed and clear the replicas.

boolean isPartialInvalidateFailure(IOException e) {
  return e.getMessage() != null && e.getMessage().startsWith("Failed to delete ")
      && e.getMessage().contains("replica(s)");
}

try {
  dataset.invalidate(bpid, blocks);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Failed to delete")) {
    // per-block causes are listed in the message; fix fs-level cause, then the NN re-issues invalidation
    logPerBlockCauses(e);
    scheduleRetry(); // NameNode will resend the delete list
  }
}

When it happens

Trigger: invalidate(bpid, invalidBlks[], async) processing a NameNode BLOCK_INVALIDATE request where one or more delete() calls fail - e.g., permission denied on the block file, I/O error from a failing disk, or path already gone in a racy way that still errored.

Common situations: Data directory permissions changed (chown/chmod by an admin); disk developing bad sectors; dfs.datanode.data.dir on NFS with stale handles; SELinux/AppArmor denying unlink.

Related errors

• Cannot remove directory {}
• Cannot remove directory {}
• Cannot create directory {}
• Possible disk error: Failed to create {}
• Cannot list contents of {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cd46ecc8cafbbb0a. Report an issue: GitHub.