apache/hadoop · error · ReplicaNotFoundException

Found fewer bytesOnDisk than bytesAcked for replica {rbw}

Error message

Found fewer bytesOnDisk than bytesAcked for replica {rbw}

What it means

recoverRbwImpl tolerates bytesOnDisk exceeding bytesAcked (the corrupt tail is truncated), never the reverse: if fewer bytes are physically on disk than were already acknowledged upstream, acked data is missing and the replica is unusable - ReplicaNotFoundException ('Found fewer bytesOnDisk than bytesAcked'). The code just above already resets bytesOnDisk from the block file's data length, so reaching this throw means the block file itself is short.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1775

      long numBytes = rbw.getNumBytes();
      if (bytesAcked < minBytesRcvd || numBytes > maxBytesRcvd) {
        throw new ReplicaNotFoundException("Unmatched length replica " +
            rbw + ": BytesAcked = " + bytesAcked +
            " BytesRcvd = " + numBytes + " are not in the range of [" +
            minBytesRcvd + ", " + maxBytesRcvd + "].");
      }

      long bytesOnDisk = rbw.getBytesOnDisk();
      long blockDataLength = rbw.getReplicaInfo().getBlockDataLength();
      if (bytesOnDisk != blockDataLength) {
        LOG.info("Resetting bytesOnDisk to match blockDataLength (={}) for " +
            "replica {}", blockDataLength, rbw);
        bytesOnDisk = blockDataLength;
        rbw.setLastChecksumAndDataLen(bytesOnDisk, null);
      }

      if (bytesOnDisk < bytesAcked) {
        throw new ReplicaNotFoundException("Found fewer bytesOnDisk than " +
            "bytesAcked for replica " + rbw);
      }

      FsVolumeReference ref = rbw.getReplicaInfo()
          .getVolume().obtainReference();
      try {
        // Truncate the potentially corrupt portion.
        // If the source was client and the last node in the pipeline was lost,
        // any corrupt data written after the acked length can go unnoticed.
        if (bytesOnDisk > bytesAcked) {
          rbw.getReplicaInfo().truncateBlock(bytesAcked);
          rbw.setNumBytes(bytesAcked);
          rbw.setLastChecksumAndDataLen(bytesAcked, null);
        }

        // bump the replica's generation stamp to newGS
        rbw.getReplicaInfo().bumpReplicaGS(newGS);
      } catch (IOException e) {

View on GitHub (pinned to 2add963021)

Solutions

Invalidate the replica on this DN - the NN re-replicates the acked range from surviving replicas
Check disk/filesystem integrity on that DataNode (volume fsck, SMART); a disk that produced one short replica will likely produce more
Run 'hdfs fsck' to confirm healthy replicas exist before removing this one
If no healthy replica covers the acked range, data is lost: restore from snapshot/backup

Defensive patterns

Strategy: try-catch

Validate before calling

ReplicaInfo raw = (ReplicaInfo) fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (raw instanceof ReplicaInPipeline) {
  ReplicaInPipeline rbw = (ReplicaInPipeline) raw;
  if (rbw.getReplicaInfo().getBlockDataLength() < rbw.getBytesAcked()) {
    markReplicaCorruptAndInvalidate(b); // acked data is physically missing
    return;
  }
}
fsDataset.recoverRbw(b, newGS, minBytesRcvd, maxBytesRcvd);

Try / catch

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage() != null && rnfe.getMessage().contains("fewer bytesOnDisk")) {
    markReplicaCorruptAndInvalidate(b); // never retry recovery against this replica
  } else { throw rnfe; }
}

Prevention

Never retry recovery against a replica that acked more than it holds - report it corrupt
Use reliable disks and filesystems with flushed metadata for DataNode storage
Watch DataNode scans/logs for short block files (bytesOnDisk < bytesAcked) as an early corruption signal

When it happens

Trigger: An RBW replica whose block file lost data relative to its ack state: truncated block file after disk failure, writes lost in page cache on a hard crash, or block file and meta restored inconsistently.

Common situations: Hard power loss or filesystem corruption on a DataNode mid-write; block file restored from backup with a newer meta file; custom storage backends losing writes.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1e1563f61d803483. Report an issue: GitHub.