{"record":{"id":"43e7e8fc98af25b5","repo":"apache/hadoop","slug":"this-is-not-supposed-to-happen-replica-getbyteson","errorCode":null,"errorMessage":"THIS IS NOT SUPPOSED TO HAPPEN: replica.getBytesOnDisk() != block.getNumBytes(), block={block}, replica={replica}","messagePattern":"THIS IS NOT SUPPOSED TO HAPPEN: replica\\.getBytesOnDisk\\(\\) != block\\.getNumBytes\\(\\), block=(.+?), replica=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3171,"sourceCode":"      LOG.info(\"updateReplica: \" + oldBlock\n          + \", recoveryId=\" + recoveryId\n          + \", length=\" + newlength\n          + \", replica=\" + replica);\n\n      //check replica\n      if (replica == null) {\n        throw new ReplicaNotFoundException(oldBlock);\n      }\n\n      //check replica state\n      if (replica.getState() != ReplicaState.RUR) {\n        throw new IOException(\"replica.getState() != \" + ReplicaState.RUR\n            + \", replica=\" + replica);\n      }\n\n      //check replica's byte on disk\n      if (replica.getBytesOnDisk() != oldBlock.getNumBytes()) {\n        throw new IOException(\"THIS IS NOT SUPPOSED TO HAPPEN:\"\n            + \" replica.getBytesOnDisk() != block.getNumBytes(), block=\"\n            + oldBlock + \", replica=\" + replica);\n      }\n\n      //check replica files before update\n      checkReplicaFiles(replica);\n\n      //update replica\n      final ReplicaInfo finalized = updateReplicaUnderRecovery(oldBlock\n          .getBlockPoolId(), replica, recoveryId,\n          newBlockId, newlength);\n\n      boolean copyTruncate = newBlockId != oldBlock.getBlockId();\n      if (!copyTruncate) {\n        assert finalized.getBlockId() == oldBlock.getBlockId()\n            && finalized.getGenerationStamp() == recoveryId\n            && finalized.getNumBytes() == newlength\n            : \"Replica information mismatched: oldBlock=\" + oldBlock","sourceCodeStart":3153,"sourceCodeEnd":3189,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3153-L3189","documentation":"Sanity check in FsDatasetImpl.updateReplica during block recovery: the bytes the replica reports on disk must equal the block length the NameNode recorded when scheduling recovery (oldBlock.getNumBytes). A mismatch means this DataNode's on-disk replica no longer matches the NameNode's committed block record, so recovery cannot safely proceed.","triggerScenarios":"Block file truncated or extended on disk by a failing drive or external tool between block report and recovery; a previous recovery committed a different length than what is physically on disk; balancer or scanner replacing the replica mid-recovery; block file restored from backup with a different length.","commonSituations":"Failing or corrupt disk on one replica; NFS-backed volumes with partial writes; manually restored block directories; recovery length races fixed in later Hadoop point releases.","solutions":["Run 'hdfs fsck /path -files -blocks -locations' to find which replica diverged and whether the block is recoverable from other replicas.","Compare the on-disk block file size under <volume>/current/<bpid>/finalized/... with the NameNode's block length from fsck output.","Delete the divergent replica (block file plus its .meta) so the NameNode re-replicates from healthy copies; DirectoryScanner reconciles the volume map.","Check disk health (dmesg, SMART) and evict the bad volume using dfs.datanode.failed.volumes.tolerated.","Upgrade to the latest point release; several block-recovery length races were fixed after HDFS-9187."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Replica r = fsDataset.getReplica(bpid, block.getBlockId());\nif (r != null && r.getBytesOnDisk() != block.getNumBytes()) {\n  // replica diverged from NN record; quarantine it instead of recovering\n  throw new IOException(\"replica length mismatch for \" + block);\n}","typeGuard":null,"tryCatchPattern":"try {\n  fsDataset.updateReplica(block, recoveryId, newlength);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"getBytesOnDisk()\")) {\n    // treat as corrupt replica: report to NN so it is invalidated and re-replicated\n  }\n}","preventionTips":["Enable the DirectoryScanner (dfs.datanode.directoryscan.interval) so volume map and disk stay reconciled.","Monitor disk health and set dfs.datanode.failed.volumes.tolerated to evict bad volumes early.","Schedule periodic 'hdfs fsck' to catch under-replicated or corrupt replicas before recovery needs them."],"tags":["hdfs","datanode","block-recovery","data-corruption","disk"],"backgroundTag":"hdfs-replica-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}