{"record":{"id":"2a4018595b306ae0","repo":"apache/hadoop","slug":"unmatched-length-replica-rbw-bytesacked-byte","errorCode":null,"errorMessage":"Unmatched length replica {rbw}: BytesAcked = {bytesAcked} BytesRcvd = {numBytes} are not in the range of [{minBytesRcvd}, {maxBytesRcvd}].","messagePattern":"Unmatched length replica (.+?): BytesAcked = (.+?) BytesRcvd = (.+?) are not in the range of \\[(.+?), (.+?)\\]\\.","errorType":"exception","errorClass":"ReplicaNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1759,"sourceCode":"      throws IOException {\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n        b.getBlockPoolId(), getStorageUuidForLock(b),\n        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n      // check generation stamp\n      long replicaGenerationStamp = rbw.getGenerationStamp();\n      if (replicaGenerationStamp < b.getGenerationStamp() ||\n          replicaGenerationStamp > newGS) {\n        throw new ReplicaNotFoundException(\n            ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + b +\n                \". Expected GS range is [\" + b.getGenerationStamp() + \", \" +\n                newGS + \"].\");\n      }\n\n      // check replica length\n      long bytesAcked = rbw.getBytesAcked();\n      long numBytes = rbw.getNumBytes();\n      if (bytesAcked < minBytesRcvd || numBytes > maxBytesRcvd) {\n        throw new ReplicaNotFoundException(\"Unmatched length replica \" +\n            rbw + \": BytesAcked = \" + bytesAcked +\n            \" BytesRcvd = \" + numBytes + \" are not in the range of [\" +\n            minBytesRcvd + \", \" + maxBytesRcvd + \"].\");\n      }\n\n      long bytesOnDisk = rbw.getBytesOnDisk();\n      long blockDataLength = rbw.getReplicaInfo().getBlockDataLength();\n      if (bytesOnDisk != blockDataLength) {\n        LOG.info(\"Resetting bytesOnDisk to match blockDataLength (={}) for \" +\n            \"replica {}\", blockDataLength, rbw);\n        bytesOnDisk = blockDataLength;\n        rbw.setLastChecksumAndDataLen(bytesOnDisk, null);\n      }\n\n      if (bytesOnDisk < bytesAcked) {\n        throw new ReplicaNotFoundException(\"Found fewer bytesOnDisk than \" +\n            \"bytesAcked for replica \" + rbw);\n      }","sourceCodeStart":1741,"sourceCodeEnd":1777,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1741-L1777","documentation":"During block recovery, the pipeline agrees on an acceptable byte range [minBytesRcvd, maxBytesRcvd] across replicas. recoverRbwImpl enforces that this DataNode's bytesAcked is at least the minimum and its numBytes (bytesRcvd) at most the maximum; otherwise the replica's length is outside the consensus and ReplicaNotFoundException ('Unmatched length replica') is thrown.","triggerScenarios":"recoverRbw on a DN whose RBW replica received more bytes than the others (numBytes > maxBytesRcvd, e.g. it kept accepting packets after the rest of the pipeline stalled) or acked fewer than the agreed minimum (bytesAcked < minBytesRcvd).","commonSituations":"Divergent pipelines after a DN crash mid-stream: replicas disagree on how much of the tail landed; recovery excludes the mismatched replica and the client rebuilds the pipeline.","solutions":["Retry - NameNode block recovery recomputes the ranges; the divergent replica is truncated or invalidated and the write continues","If the same DN keeps failing, invalidate its replica and let re-replication heal it","Check for asymmetric packet loss or long GC/IO pauses on the failing DN (sources of divergent tails)","fsck the file once the write completes"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"ReplicaInfo raw = (ReplicaInfo) fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (raw instanceof ReplicaInPipeline) {\n  ReplicaInPipeline rbw = (ReplicaInPipeline) raw;\n  if (rbw.getBytesAcked() < minBytesRcvd || rbw.getNumBytes() > maxBytesRcvd) {\n    excludeReplicaFromPipeline(b); // outside the consensus range\n    return;\n  }\n}\nfsDataset.recoverRbw(b, newGS, minBytesRcvd, maxBytesRcvd);","typeGuard":"boolean lengthWithinConsensus(ReplicaInPipeline rbw, long minBytesRcvd, long maxBytesRcvd) {\n  return rbw.getBytesAcked() >= minBytesRcvd && rbw.getNumBytes() <= maxBytesRcvd;\n}","tryCatchPattern":"catch (ReplicaNotFoundException rnfe) {\n  if (rnfe.getMessage() != null && rnfe.getMessage().startsWith(\"Unmatched length replica\")) {\n    rebuildPipelineExcludingThisReplica(); // NN recomputes the consensus range\n  } else { throw rnfe; }\n}","preventionTips":["Exclude length-divergent replicas from the pipeline instead of retrying against the same range","Recover stalled pipelines promptly so replica tails cannot diverge further","Monitor DN GC and IO pauses - asymmetric packet delivery creates unmatched tails"],"tags":["hdfs","datanode","block-recovery","rbw","length-mismatch"],"backgroundTag":"hdfs-replica-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}