{"record":{"id":"64174b9d194bb777","repo":"apache/hadoop","slug":"corrupted-replica-replicainfo-with-a-length-of-64174b","errorCode":null,"errorMessage":"Corrupted replica {replicaInfo} with a length of {numBytes} expected length is {expectedBlockLen}","messagePattern":"Corrupted replica (.+?) with a length of (.+?) expected length is (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1424,"sourceCode":"      // If the block was successfully finalized because all packets\n      // were successfully processed at the Datanode but the ack for\n      // some of the packets were not received by the client. The client\n      // re-opens the connection and retries sending those packets.\n      // The other reason is that an \"append\" is occurring to this block.\n\n      // check the validity of the parameter\n      if (newGS < b.getGenerationStamp()) {\n        throw new IOException(\"The new generation stamp \" + newGS +\n            \" should be greater than the replica \" + b + \"'s generation stamp\");\n      }\n      ReplicaInfo replicaInfo = getReplicaInfo(b);\n      LOG.info(\"Appending to \" + replicaInfo);\n      if (replicaInfo.getState() != ReplicaState.FINALIZED) {\n        throw new ReplicaNotFoundException(\n            ReplicaNotFoundException.UNFINALIZED_REPLICA + b);\n      }\n      if (replicaInfo.getNumBytes() != expectedBlockLen) {\n        throw new IOException(\"Corrupted replica \" + replicaInfo +\n            \" with a length of \" + replicaInfo.getNumBytes() +\n            \" expected length is \" + expectedBlockLen);\n      }\n\n      FsVolumeReference ref = replicaInfo.getVolume().obtainReference();\n      ReplicaInPipeline replica = null;\n      try {\n        replica = append(b.getBlockPoolId(), replicaInfo, newGS,\n            b.getNumBytes());\n      } catch (IOException e) {\n        IOUtils.cleanupWithLogger(null, ref);\n        throw e;\n      }\n      return new ReplicaHandler(replica, ref);\n    }\n  }\n  \n  /** Append to a finalized replica","sourceCodeStart":1406,"sourceCodeEnd":1442,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1406-L1442","documentation":"During append, FsDatasetImpl compares the replica's stored length (getNumBytes) with expectedBlockLen, the committed block length the caller supplies. A mismatch means this DataNode's copy of the block does not match the committed block, so it is treated as corrupted and the append is refused with an IOException.","triggerScenarios":"data.append(b, newGS, expectedBlockLen) where replicaInfo.getNumBytes() != expectedBlockLen: the replica was truncated or extended relative to the committed block, e.g. a lost flush, manual damage to the block file, or block file/meta restored out of sync after a disk incident.","commonSituations":"Corrupt or partially lost block file after disk failure; replica inconsistent after a botched recovery; block restored from backup with a different length; NameNode block info diverging from one DataNode's storage.","solutions":["Run 'hdfs fsck <file> -files -blocks -locations' to confirm which replica's length diverges","Remove/invalidate the corrupt replica (hdfs debug invalidateBlocks, or let the NN mark it corrupt) so the healthy replicas with the correct length win and re-replication heals","Check DataNode disk/filesystem health (dmesg, SMART) - silent corruption is often the root cause","If every replica disagrees with expectedBlockLen, committed data is lost: restore from snapshot/backup or accept loss via fsck -delete and rewrite"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (r != null && r.getNumBytes() != expectedBlockLen) {\n  throw new IOException(\"Refusing append: local replica length \" + r.getNumBytes()\n      + \" != committed \" + expectedBlockLen + \"; replica is corrupt\");\n}\nfsDataset.append(b, newGS, expectedBlockLen);","typeGuard":"boolean matchesCommittedLength(FsDatasetSpi data, ExtendedBlock b, long expectedLen) throws IOException {\n  Replica r = data.getReplica(b.getBlockPoolId(), b.getBlockId());\n  return r != null && r.getNumBytes() == expectedLen;\n}","tryCatchPattern":"catch (IOException ioe) {\n  if (ioe.getMessage() != null && ioe.getMessage().startsWith(\"Corrupted replica\")) {\n    reportReplicaCorruptToNameNode(b); // trigger re-replication from healthy copies\n  } else { throw ioe; }\n}","preventionTips":["Treat any length mismatch as corruption: report the replica to the NameNode so a healthy one is used","Run fsck periodically to catch length-divergent replicas before a user appends","Keep DataNode disks monitored (SMART, directory scanner) to catch bit rot early"],"tags":["hdfs","datanode","append","replica-corruption","length-mismatch"],"backgroundTag":"hdfs-replica-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}