apache/hadoop · error · ReplicaNotFoundException

Unmatched length replica {rbw}: BytesAcked = {bytesAcked} By

Error message

Unmatched length replica {rbw}: BytesAcked = {bytesAcked} BytesRcvd = {numBytes} are not in the range of [{minBytesRcvd}, {maxBytesRcvd}].

What it means

During block recovery, the pipeline agrees on an acceptable byte range [minBytesRcvd, maxBytesRcvd] across replicas. recoverRbwImpl enforces that this DataNode's bytesAcked is at least the minimum and its numBytes (bytesRcvd) at most the maximum; otherwise the replica's length is outside the consensus and ReplicaNotFoundException ('Unmatched length replica') is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1759

      throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        b.getBlockPoolId(), getStorageUuidForLock(b),
        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
      // check generation stamp
      long replicaGenerationStamp = rbw.getGenerationStamp();
      if (replicaGenerationStamp < b.getGenerationStamp() ||
          replicaGenerationStamp > newGS) {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + b +
                ". Expected GS range is [" + b.getGenerationStamp() + ", " +
                newGS + "].");
      }

      // check replica length
      long bytesAcked = rbw.getBytesAcked();
      long numBytes = rbw.getNumBytes();
      if (bytesAcked < minBytesRcvd || numBytes > maxBytesRcvd) {
        throw new ReplicaNotFoundException("Unmatched length replica " +
            rbw + ": BytesAcked = " + bytesAcked +
            " BytesRcvd = " + numBytes + " are not in the range of [" +
            minBytesRcvd + ", " + maxBytesRcvd + "].");
      }

      long bytesOnDisk = rbw.getBytesOnDisk();
      long blockDataLength = rbw.getReplicaInfo().getBlockDataLength();
      if (bytesOnDisk != blockDataLength) {
        LOG.info("Resetting bytesOnDisk to match blockDataLength (={}) for " +
            "replica {}", blockDataLength, rbw);
        bytesOnDisk = blockDataLength;
        rbw.setLastChecksumAndDataLen(bytesOnDisk, null);
      }

      if (bytesOnDisk < bytesAcked) {
        throw new ReplicaNotFoundException("Found fewer bytesOnDisk than " +
            "bytesAcked for replica " + rbw);
      }

View on GitHub (pinned to 2add963021)

Solutions

Retry - NameNode block recovery recomputes the ranges; the divergent replica is truncated or invalidated and the write continues
If the same DN keeps failing, invalidate its replica and let re-replication heal it
Check for asymmetric packet loss or long GC/IO pauses on the failing DN (sources of divergent tails)
fsck the file once the write completes

Defensive patterns

Strategy: validation

Validate before calling

ReplicaInfo raw = (ReplicaInfo) fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (raw instanceof ReplicaInPipeline) {
  ReplicaInPipeline rbw = (ReplicaInPipeline) raw;
  if (rbw.getBytesAcked() < minBytesRcvd || rbw.getNumBytes() > maxBytesRcvd) {
    excludeReplicaFromPipeline(b); // outside the consensus range
    return;
  }
}
fsDataset.recoverRbw(b, newGS, minBytesRcvd, maxBytesRcvd);

Type guard

boolean lengthWithinConsensus(ReplicaInPipeline rbw, long minBytesRcvd, long maxBytesRcvd) {
  return rbw.getBytesAcked() >= minBytesRcvd && rbw.getNumBytes() <= maxBytesRcvd;
}

Try / catch

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage() != null && rnfe.getMessage().startsWith("Unmatched length replica")) {
    rebuildPipelineExcludingThisReplica(); // NN recomputes the consensus range
  } else { throw rnfe; }
}

Prevention

Exclude length-divergent replicas from the pipeline instead of retrying against the same range
Recover stalled pipelines promptly so replica tails cannot diverge further
Monitor DN GC and IO pauses - asymmetric packet delivery creates unmatched tails

When it happens

Trigger: recoverRbw on a DN whose RBW replica received more bytes than the others (numBytes > maxBytesRcvd, e.g. it kept accepting packets after the rest of the pipeline stalled) or acked fewer than the agreed minimum (bytesAcked < minBytesRcvd).

Common situations: Divergent pipelines after a DN crash mid-stream: replicas disagree on how much of the tail landed; recovery excludes the mismatched replica and the client rebuilds the pipeline.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2a4018595b306ae0. Report an issue: GitHub.