apache/hadoop · error · IOException

Corrupted replica {replicaInfo} with a length of {replicaLen

Error message

Corrupted replica {replicaInfo} with a length of {replicaLen} expected length is {expectedBlockLen}

What it means

The final step of recoverCheck: after state and generation-stamp checks, the replica's length must equal expectedBlockLen, the block length this recovery is committing. A mismatch means this copy does not match the block being recovered, so it is treated as corrupted and an IOException is thrown.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1540

    long replicaLen = replicaInfo.getNumBytes();
    if (replicaInfo.getState() == ReplicaState.RBW) {
      ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;
      if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {
        throw new MustStopExistingWriter(rbw);
      }
      // check length: bytesRcvd, bytesOnDisk, and bytesAcked should be the same
      if (replicaLen != rbw.getBytesOnDisk() 
          || replicaLen != rbw.getBytesAcked()) {
        throw new ReplicaAlreadyExistsException("RBW replica " + replicaInfo + 
            "bytesRcvd(" + rbw.getNumBytes() + "), bytesOnDisk(" + 
            rbw.getBytesOnDisk() + "), and bytesAcked(" + rbw.getBytesAcked() +
            ") are not the same.");
      }
    }
    
    // check block length
    if (replicaLen != expectedBlockLen) {
      throw new IOException("Corrupted replica " + replicaInfo + 
          " with a length of " + replicaLen + 
          " expected length is " + expectedBlockLen);
    }
    
    return replicaInfo;
  }

  @Override  // FsDatasetSpi
  public ReplicaHandler recoverAppend(
      ExtendedBlock b, long newGS, long expectedBlockLen) throws IOException {
    LOG.info("Recover failed append to " + b);

    while (true) {
      try {
        try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
            b.getBlockPoolId(), getStorageUuidForLock(b),
            datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
          ReplicaInfo replicaInfo = recoverCheck(b, newGS, expectedBlockLen);

View on GitHub (pinned to 2add963021)

Solutions

1. Let recovery proceed on the other replicas and invalidate this one; confirm with 'hdfs fsck <file> -files -blocks -locations'
2. Remove the corrupt replica so the NN schedules re-replication
3. Verify expectedBlockLen passed by the recovery caller matches the NN block length - a stale caller produces false corruption errors
4. Investigate disk health on the DataNode reporting the mismatch

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (r != null && r.getNumBytes() != expectedBlockLen) {
  reportReplicaCorruptToNameNode(b); // let re-replication handle it
  return;
}
fsDataset.recoverAppend(b, newGS, expectedBlockLen);

boolean matchesExpectedLength(Replica r, long expectedBlockLen) {
  return r != null && r.getNumBytes() == expectedBlockLen;
}

catch (IOException ioe) {
  if (ioe.getMessage() != null && ioe.getMessage().startsWith("Corrupted replica")) {
    reportReplicaCorruptToNameNode(b); // never retry recovery against this replica
  } else { throw ioe; }
}

When it happens

Trigger: recoverAppend/recoverClose where replicaLen (getNumBytes) != expectedBlockLen supplied by the recovery coordinator - a truncated or extended replica relative to the committed block, or a coordinator passing a stale length.

Common situations: Corrupt replica after disk trouble; recovery caller computing expectedBlockLen from a different, diverged replica; recovery racing a concurrent write that changed the replica length.

Related errors

• Corrupted replica {replicaInfo} with a length of {numBytes}
• RBW replica {replicaInfo}bytesRcvd({numBytes}), bytesOnDisk(
• Unmatched length replica {rbw}: BytesAcked = {bytesAcked} By
• Found fewer bytesOnDisk than bytesAcked for replica {rbw}
• Replica was found but missing fields.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/38271cbf33dd8fa1. Report an issue: GitHub.