apache/hadoop · error · IOException

Corrupted replica {replicaInfo} with a length of {numBytes}

Error message

Corrupted replica {replicaInfo} with a length of {numBytes} expected length is {numBytes}

What it means

Thrown as plain IOException from FsDatasetImpl.moveBlockAcrossStorage when replicaInfo.getNumBytes() differs from block.getNumBytes(). The NameNode-supplied ExtendedBlock is the authority on expected length; a local replica with a different on-disk length indicates the replica is stale, truncated, or from a different write generation, and copying/moving it would propagate corruption.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1092

    return new File[]{dstMeta, dstFile};
  }

  /**
   * Move block files from one storage to another storage.
   * @return Returns the Old replicaInfo
   * @throws IOException
   */
  @Override
  public ReplicaInfo moveBlockAcrossStorage(ExtendedBlock block,
      StorageType targetStorageType, String targetStorageId)
      throws IOException {
    ReplicaInfo replicaInfo = getReplicaInfo(block);
    if (replicaInfo.getState() != ReplicaState.FINALIZED) {
      throw new ReplicaNotFoundException(
          ReplicaNotFoundException.UNFINALIZED_REPLICA + block);
    }
    if (replicaInfo.getNumBytes() != block.getNumBytes()) {
      throw new IOException("Corrupted replica " + replicaInfo
          + " with a length of " + replicaInfo.getNumBytes()
          + " expected length is " + block.getNumBytes());
    }
    if (replicaInfo.getVolume().getStorageType() == targetStorageType) {
      throw new ReplicaAlreadyExistsException("Replica " + replicaInfo
          + " already exists on storage " + targetStorageType);
    }

    if (replicaInfo.isOnTransientStorage()) {
      // Block movement from RAM_DISK will be done by LazyPersist mechanism
      throw new IOException("Replica " + replicaInfo
          + " cannot be moved from storageType : "
          + replicaInfo.getVolume().getStorageType());
    }

    FsVolumeReference volumeRef = null;
    boolean shouldConsiderSameMountVolume =
        shouldConsiderSameMountVolume(replicaInfo.getVolume(),

View on GitHub (pinned to 2add963021)

Solutions

Validate the file: hdfs fsck <file> -files -blocks -locations; if other replicas are healthy, delete/evict the mismatched replica (hdfs debug evacuateBlock or let the scanner invalidate it) and let re-replication replace it.
Compare lengths: the message prints replica length vs expected — a smaller local length is classic truncation.
Do not retry the move for this replica; the mismatch will not self-heal and retrying just re-throws.
If all replicas mismatch, restore the file from snapshot/backup — the NameNode's expected length is unrecoverable locally.

Defensive patterns

Strategy: validation

Validate before calling

// Compare lengths before moving; mismatch means corruption, never retry.
ReplicaInfo info = fsDataset.getReplicaInfo(
    block.getBlockPoolId(), block.getBlockId());
if (info.getNumBytes() != block.getNumBytes()) {
  reportCorruptReplica(block, info.getNumBytes());
  continue; // do NOT move a length-mismatched replica
}

Try / catch

// Length mismatch is non-retryable: quarantine replica, re-replicate.
try {
  fsDataset.moveBlockAcrossStorage(block, target, null);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).startsWith("Corrupted replica")) {
    reportCorruptToNameNode(dfsClient, block, e);
    return; // re-replication replaces it; retrying the move is wrong
  }
  throw e;
}

Prevention

Always validate replica length against the NameNode's expected length before storage moves.
On mismatch, report a corrupt replica (dfsClient.reportCorruptReplica) instead of retrying.
Monitor disk health (SMART, DN volume metrics) — truncation is usually a dying-disk symptom.

When it happens

Trigger: A storage-policy move requested for a replica whose recorded length disagrees with the NameNode's block length: truncated by a dying disk, stale after append recovery bumped length elsewhere, replica resurrected from a bad volume, or race between an updateBlock and the move request.

Common situations: Bit-rot/truncation on one DN while other replicas are healthy; move racing an append/lease recovery that changed the block length; restored-from-backup data dirs with older replica generations.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6c862a9702a3ed0a. Report an issue: GitHub.