{"record":{"id":"33ea299484cc4fb0","repo":"apache/hadoop","slug":"replica-is-not-readable-block-block-replica-r","errorCode":null,"errorMessage":"Replica is not readable, block={block}, replica={replica}","messagePattern":"Replica is not readable, block=(.+?), replica=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java","lineNumber":286,"sourceCode":"      }\n      if (replica instanceof FinalizedReplica) {\n        chunkChecksum = getPartialChunkChecksumForFinalized(\n            (FinalizedReplica)replica);\n      }\n\n      if (replica.getGenerationStamp() < block.getGenerationStamp()) {\n        throw new IOException(\"Replica gen stamp < block genstamp, block=\"\n            + block + \", replica=\" + replica);\n      } else if (replica.getGenerationStamp() > block.getGenerationStamp()) {\n        if (DataNode.LOG.isDebugEnabled()) {\n          DataNode.LOG.debug(\"Bumping up the client provided\"\n              + \" block's genstamp to latest \" + replica.getGenerationStamp()\n              + \" for block \" + block);\n        }\n        block.setGenerationStamp(replica.getGenerationStamp());\n      }\n      if (replicaVisibleLength < 0) {\n        throw new IOException(\"Replica is not readable, block=\"\n            + block + \", replica=\" + replica);\n      }\n      if (DataNode.LOG.isDebugEnabled()) {\n        DataNode.LOG.debug(\"block=\" + block + \", replica=\" + replica);\n      }\n\n      // transferToFully() fails on 32 bit platforms for block sizes >= 2GB,\n      // use normal transfer in those cases\n      this.transferToAllowed = datanode.getDnConf().transferToAllowed &&\n        (!is32Bit || length <= Integer.MAX_VALUE);\n\n      // Obtain a reference before reading data\n      FsVolumeSpi volume = datanode.data.getVolume(block);\n      if (volume == null) {\n        LOG.warn(\"Cannot find FsVolumeSpi to obtain a reference for block: {}\", block);\n        throw new ReplicaNotFoundException(block);\n      }\n      volumeRef = volume.obtainReference();","sourceCodeStart":268,"sourceCodeEnd":304,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java#L268-L304","documentation":"BlockSender computes replicaVisibleLength from the replica; a negative value means the replica exposes no readable bytes — the replica's visible length is undefined (for example an RBW replica whose state/length was never published, or one the volume/replica manager considers unreadable). The read is refused before any byte is served.","triggerScenarios":"openBlock on a replica whose getVisibleLength()/state yields a negative visible length: reads against a torn-down RBW replica after a failed pipeline, a replica mid-recovery whose visible length was reset, or a corrupted replica descriptor in FsDatasetImpl.","commonSituations":"Reading a block that a writer just abandoned; a datanode recovering from a crash where RBW replicas were left in an unpublishable state; short-circuit or normal reads racing with replica deletion/recovery.","solutions":["Retry the read on a different replica: the client's BlockReaderFactory/DFSInputStream falls back to another location, which usually resolves the read.","On the DN, check the replica state in logs (RBW/RWR/RR/FINALIZED) for this block id; a stuck non-finalized replica after crash recovery should be re-recovered or discarded.","If every replica reports unreadable, use fsck to locate and repair/re-replicate the block.","Ensure datanodes were restarted cleanly after crashes so leftover RBW replicas get converted by recovery-on-startup."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  sender = new BlockSender(block, startOffset, length, ...);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Replica is not readable\")) {\n    fallbackToNextReplica(); // reader pool / DFSInputStream style failover\n  } else {\n    throw e;\n  }\n}","preventionTips":["Restart datanodes cleanly after crashes so orphaned RBW replicas get recovered into readable states.","Read via DFSInputStream rather than raw data-transfer APIs so replica failover is automatic.","Alert when a replica repeatedly reports unreadable — it usually precedes replica-state corruption."],"tags":["hdfs","datanode","read-path","replica-state","rbw"],"backgroundTag":"unreadable-replica","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}