{"record":{"id":"38271cbf33dd8fa1","repo":"apache/hadoop","slug":"corrupted-replica-replicainfo-with-a-length-of-38271c","errorCode":null,"errorMessage":"Corrupted replica {replicaInfo} with a length of {replicaLen} expected length is {expectedBlockLen}","messagePattern":"Corrupted replica (.+?) with a length of (.+?) expected length is (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1540,"sourceCode":"    long replicaLen = replicaInfo.getNumBytes();\n    if (replicaInfo.getState() == ReplicaState.RBW) {\n      ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;\n      if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {\n        throw new MustStopExistingWriter(rbw);\n      }\n      // check length: bytesRcvd, bytesOnDisk, and bytesAcked should be the same\n      if (replicaLen != rbw.getBytesOnDisk() \n          || replicaLen != rbw.getBytesAcked()) {\n        throw new ReplicaAlreadyExistsException(\"RBW replica \" + replicaInfo + \n            \"bytesRcvd(\" + rbw.getNumBytes() + \"), bytesOnDisk(\" + \n            rbw.getBytesOnDisk() + \"), and bytesAcked(\" + rbw.getBytesAcked() +\n            \") are not the same.\");\n      }\n    }\n    \n    // check block length\n    if (replicaLen != expectedBlockLen) {\n      throw new IOException(\"Corrupted replica \" + replicaInfo + \n          \" with a length of \" + replicaLen + \n          \" expected length is \" + expectedBlockLen);\n    }\n    \n    return replicaInfo;\n  }\n\n  @Override  // FsDatasetSpi\n  public ReplicaHandler recoverAppend(\n      ExtendedBlock b, long newGS, long expectedBlockLen) throws IOException {\n    LOG.info(\"Recover failed append to \" + b);\n\n    while (true) {\n      try {\n        try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n            b.getBlockPoolId(), getStorageUuidForLock(b),\n            datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n          ReplicaInfo replicaInfo = recoverCheck(b, newGS, expectedBlockLen);","sourceCodeStart":1522,"sourceCodeEnd":1558,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1522-L1558","documentation":"The final step of recoverCheck: after state and generation-stamp checks, the replica's length must equal expectedBlockLen, the block length this recovery is committing. A mismatch means this copy does not match the block being recovered, so it is treated as corrupted and an IOException is thrown.","triggerScenarios":"recoverAppend/recoverClose where replicaLen (getNumBytes) != expectedBlockLen supplied by the recovery coordinator - a truncated or extended replica relative to the committed block, or a coordinator passing a stale length.","commonSituations":"Corrupt replica after disk trouble; recovery caller computing expectedBlockLen from a different, diverged replica; recovery racing a concurrent write that changed the replica length.","solutions":["Let recovery proceed on the other replicas and invalidate this one; confirm with 'hdfs fsck <file> -files -blocks -locations'","Remove the corrupt replica so the NN schedules re-replication","Verify expectedBlockLen passed by the recovery caller matches the NN block length - a stale caller produces false corruption errors","Investigate disk health on the DataNode reporting the mismatch"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (r != null && r.getNumBytes() != expectedBlockLen) {\n  reportReplicaCorruptToNameNode(b); // let re-replication handle it\n  return;\n}\nfsDataset.recoverAppend(b, newGS, expectedBlockLen);","typeGuard":"boolean matchesExpectedLength(Replica r, long expectedBlockLen) {\n  return r != null && r.getNumBytes() == expectedBlockLen;\n}","tryCatchPattern":"catch (IOException ioe) {\n  if (ioe.getMessage() != null && ioe.getMessage().startsWith(\"Corrupted replica\")) {\n    reportReplicaCorruptToNameNode(b); // never retry recovery against this replica\n  } else { throw ioe; }\n}","preventionTips":["Cross-check expectedBlockLen against the NameNode's block info before starting recovery","Report length-mismatched replicas as corrupt rather than retrying recovery against them","Watch fsck output for length divergence between replicas of the same block"],"tags":["hdfs","datanode","block-recovery","replica-corruption","length-mismatch"],"backgroundTag":"hdfs-replica-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}