apache/hadoop · warning · ReplicaNotFoundException

Replica not found for {block}. The block may have been remov

Error message

Replica not found for {block}. The block may have been removed recently by the balancer or by intentionally reducing the replication factor. This condition is usually harmless. To be certain, please check the preceding datanode log messages for signs of a more serious issue.

What it means

Before reading, BlockSender must pin the replica's volume via datanode.data.getVolume(block); a null FsVolumeSpi means no volume in the dataset currently owns this block, so ReplicaNotFoundException is thrown. The message itself states the common truth: the block was removed (balancer move, replication-factor reduction, deletion) after the NameNode handed out its location, and the condition is usually harmless — the client just needs fresher locations.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:302

      }
      if (replicaVisibleLength < 0) {
        throw new IOException("Replica is not readable, block="
            + block + ", replica=" + replica);
      }
      if (DataNode.LOG.isDebugEnabled()) {
        DataNode.LOG.debug("block=" + block + ", replica=" + replica);
      }

      // transferToFully() fails on 32 bit platforms for block sizes >= 2GB,
      // use normal transfer in those cases
      this.transferToAllowed = datanode.getDnConf().transferToAllowed &&
        (!is32Bit || length <= Integer.MAX_VALUE);

      // Obtain a reference before reading data
      FsVolumeSpi volume = datanode.data.getVolume(block);
      if (volume == null) {
        LOG.warn("Cannot find FsVolumeSpi to obtain a reference for block: {}", block);
        throw new ReplicaNotFoundException(block);
      }
      volumeRef = volume.obtainReference();

      /* 
       * (corruptChecksumOK, meta_file_exist): operation
       * True,   True: will verify checksum  
       * True,  False: No verify, e.g., need to read data from a corrupted file 
       * False,  True: will verify checksum
       * False, False: throws IOException file not found
       */
      DataChecksum csum = null;
      if (verifyChecksum || sendChecksum) {
        LengthInputStream metaIn = null;
        boolean keepMetaInOpen = false;
        try {
          DataNodeFaultInjector.get().throwTooManyOpenFiles();
          metaIn = datanode.data.getMetaDataInputStream(block);
          if (!corruptChecksumOk || metaIn != null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Treat as expected noise: the client automatically re-fetches block locations from the NameNode and reads a surviving replica.
2. Verify block health once with 'hdfs fsck /file -files -blocks -locations' if the message repeats for the same block.
3. Reduce staleness in custom clients by re-resolving LocatedBlocks on ReplicaNotFoundException instead of failing the read.
4. Rule out a serious cause by checking preceding DN log lines for disk or dataset errors, as the message advises.

Example fix

// before: custom client fails on stale location
try {
  reader = new BlockReaderFactory(...).build();
} catch (ReplicaNotFoundException e) {
  throw new IOException("read failed", e);
}

// after: refetch locations and retry on another replica
try {
  reader = new BlockReaderFactory(...).build();
} catch (ReplicaNotFoundException e) {
  locatedBlocks = namenode.getBlockLocations(file, offset, len);
  reader = openReaderFrom(locatedBlocks); // retry with fresh locations
}

// Client-side: verify the DN you are about to read from is still in fresh locations
LocatedBlock lb = namenode.getBlockLocations(file, offset, 1).get(0);
boolean stillHosted = Arrays.stream(lb.getLocations())
    .anyMatch(di -> di.getXferAddr().equals(targetXferAddr));
if (!stillHosted) chooseAnotherReplica(lb);

static boolean isReplicaNotFound(IOException e) {
  return e instanceof ReplicaNotFoundException ||
      (e.getMessage() != null && e.getMessage().startsWith("Replica not found for"));
}

try {
  reader = buildBlockReader(block, targetDn);
} catch (ReplicaNotFoundException e) {
  lb = namenode.getBlockLocations(file, offset, len).get(0); // refresh locations
  reader = buildBlockReader(lb, pickReplica(lb));          // read from another DN
}

When it happens

Trigger: A client or peer DN opens a read for a block whose files were already deleted on this datanode: dfs.datanode.replication factor lowered, the balancer moved the replica off, or an HDFS delete/unlease removed it between the NN block-report and the read request.

Common situations: Reads racing with decreasing replication factor; balancer aggressively moving blocks; clients using stale located-block caches; decommissioning datanodes while reads are in flight.

Related errors

• No data exists for block {}
• Replica gen stamp < block genstamp, block={block}, replica={
• Replica is not readable, block={block}, replica={replica}
• Meta-data not found for {block}
• The meta file length {metaIn.getLength()} is less than the e

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ccb94cd94dd828e7. Report an issue: GitHub.