apache/hadoop · error · IOException

Corrupted replica {replicaInfo} with a length of {numBytes}

Error message

Corrupted replica {replicaInfo} with a length of {numBytes} expected length is {expectedBlockLen}

What it means

During append, FsDatasetImpl compares the replica's stored length (getNumBytes) with expectedBlockLen, the committed block length the caller supplies. A mismatch means this DataNode's copy of the block does not match the committed block, so it is treated as corrupted and the append is refused with an IOException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1424

      // If the block was successfully finalized because all packets
      // were successfully processed at the Datanode but the ack for
      // some of the packets were not received by the client. The client
      // re-opens the connection and retries sending those packets.
      // The other reason is that an "append" is occurring to this block.

      // check the validity of the parameter
      if (newGS < b.getGenerationStamp()) {
        throw new IOException("The new generation stamp " + newGS +
            " should be greater than the replica " + b + "'s generation stamp");
      }
      ReplicaInfo replicaInfo = getReplicaInfo(b);
      LOG.info("Appending to " + replicaInfo);
      if (replicaInfo.getState() != ReplicaState.FINALIZED) {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.UNFINALIZED_REPLICA + b);
      }
      if (replicaInfo.getNumBytes() != expectedBlockLen) {
        throw new IOException("Corrupted replica " + replicaInfo +
            " with a length of " + replicaInfo.getNumBytes() +
            " expected length is " + expectedBlockLen);
      }

      FsVolumeReference ref = replicaInfo.getVolume().obtainReference();
      ReplicaInPipeline replica = null;
      try {
        replica = append(b.getBlockPoolId(), replicaInfo, newGS,
            b.getNumBytes());
      } catch (IOException e) {
        IOUtils.cleanupWithLogger(null, ref);
        throw e;
      }
      return new ReplicaHandler(replica, ref);
    }
  }
  
  /** Append to a finalized replica

View on GitHub (pinned to 2add963021)

Solutions

1. Run 'hdfs fsck <file> -files -blocks -locations' to confirm which replica's length diverges
2. Remove/invalidate the corrupt replica (hdfs debug invalidateBlocks, or let the NN mark it corrupt) so the healthy replicas with the correct length win and re-replication heals
3. Check DataNode disk/filesystem health (dmesg, SMART) - silent corruption is often the root cause
4. If every replica disagrees with expectedBlockLen, committed data is lost: restore from snapshot/backup or accept loss via fsck -delete and rewrite

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (r != null && r.getNumBytes() != expectedBlockLen) {
  throw new IOException("Refusing append: local replica length " + r.getNumBytes()
      + " != committed " + expectedBlockLen + "; replica is corrupt");
}
fsDataset.append(b, newGS, expectedBlockLen);

boolean matchesCommittedLength(FsDatasetSpi data, ExtendedBlock b, long expectedLen) throws IOException {
  Replica r = data.getReplica(b.getBlockPoolId(), b.getBlockId());
  return r != null && r.getNumBytes() == expectedLen;
}

catch (IOException ioe) {
  if (ioe.getMessage() != null && ioe.getMessage().startsWith("Corrupted replica")) {
    reportReplicaCorruptToNameNode(b); // trigger re-replication from healthy copies
  } else { throw ioe; }
}

When it happens

Trigger: data.append(b, newGS, expectedBlockLen) where replicaInfo.getNumBytes() != expectedBlockLen: the replica was truncated or extended relative to the committed block, e.g. a lost flush, manual damage to the block file, or block file/meta restored out of sync after a disk incident.

Common situations: Corrupt or partially lost block file after disk failure; replica inconsistent after a botched recovery; block restored from backup with a different length; NameNode block info diverging from one DataNode's storage.

Related errors

• Corrupted replica {replicaInfo} with a length of {replicaLen
• Cannot append to a replica with unexpected generation stamp
• The new generation stamp {newGS} should be greater than the
• Cannot append to an unfinalized replica {b}
• Only a Finalized replica can be appended to; Replica with bl

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/64174b9d194bb777. Report an issue: GitHub.