apache/hadoop · error · IOException

Replica is not readable, block={block}, replica={replica}

Error message

Replica is not readable, block={block}, replica={replica}

What it means

BlockSender computes replicaVisibleLength from the replica; a negative value means the replica exposes no readable bytes — the replica's visible length is undefined (for example an RBW replica whose state/length was never published, or one the volume/replica manager considers unreadable). The read is refused before any byte is served.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:286

      }
      if (replica instanceof FinalizedReplica) {
        chunkChecksum = getPartialChunkChecksumForFinalized(
            (FinalizedReplica)replica);
      }

      if (replica.getGenerationStamp() < block.getGenerationStamp()) {
        throw new IOException("Replica gen stamp < block genstamp, block="
            + block + ", replica=" + replica);
      } else if (replica.getGenerationStamp() > block.getGenerationStamp()) {
        if (DataNode.LOG.isDebugEnabled()) {
          DataNode.LOG.debug("Bumping up the client provided"
              + " block's genstamp to latest " + replica.getGenerationStamp()
              + " for block " + block);
        }
        block.setGenerationStamp(replica.getGenerationStamp());
      }
      if (replicaVisibleLength < 0) {
        throw new IOException("Replica is not readable, block="
            + block + ", replica=" + replica);
      }
      if (DataNode.LOG.isDebugEnabled()) {
        DataNode.LOG.debug("block=" + block + ", replica=" + replica);
      }

      // transferToFully() fails on 32 bit platforms for block sizes >= 2GB,
      // use normal transfer in those cases
      this.transferToAllowed = datanode.getDnConf().transferToAllowed &&
        (!is32Bit || length <= Integer.MAX_VALUE);

      // Obtain a reference before reading data
      FsVolumeSpi volume = datanode.data.getVolume(block);
      if (volume == null) {
        LOG.warn("Cannot find FsVolumeSpi to obtain a reference for block: {}", block);
        throw new ReplicaNotFoundException(block);
      }
      volumeRef = volume.obtainReference();

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the read on a different replica: the client's BlockReaderFactory/DFSInputStream falls back to another location, which usually resolves the read.
2. On the DN, check the replica state in logs (RBW/RWR/RR/FINALIZED) for this block id; a stuck non-finalized replica after crash recovery should be re-recovered or discarded.
3. If every replica reports unreadable, use fsck to locate and repair/re-replicate the block.
4. Ensure datanodes were restarted cleanly after crashes so leftover RBW replicas get converted by recovery-on-startup.

try {
  sender = new BlockSender(block, startOffset, length, ...);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Replica is not readable")) {
    fallbackToNextReplica(); // reader pool / DFSInputStream style failover
  } else {
    throw e;
  }
}

When it happens

Trigger: openBlock on a replica whose getVisibleLength()/state yields a negative visible length: reads against a torn-down RBW replica after a failed pipeline, a replica mid-recovery whose visible length was reset, or a corrupted replica descriptor in FsDatasetImpl.

Common situations: Reading a block that a writer just abandoned; a datanode recovering from a crash where RBW replicas were left in an unpublishable state; short-circuit or normal reads racing with replica deletion/recovery.

Related errors

• Cannot recover a non-RBW replica {replicaInfo}
• Replica gen stamp < block genstamp, block={block}, replica={
• Replica not found for {block}. The block may have been remov
• Meta-data not found for {block}
• The meta file length {metaIn.getLength()} is less than the e

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/33ea299484cc4fb0. Report an issue: GitHub.