apache/hadoop · critical · IOException

THIS IS NOT SUPPOSED TO HAPPEN: replica.getBytesOnDisk() !=

Error message

THIS IS NOT SUPPOSED TO HAPPEN: replica.getBytesOnDisk() != block.getNumBytes(), block={block}, replica={replica}

What it means

Sanity check in FsDatasetImpl.updateReplica during block recovery: the bytes the replica reports on disk must equal the block length the NameNode recorded when scheduling recovery (oldBlock.getNumBytes). A mismatch means this DataNode's on-disk replica no longer matches the NameNode's committed block record, so recovery cannot safely proceed.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3171

      LOG.info("updateReplica: " + oldBlock
          + ", recoveryId=" + recoveryId
          + ", length=" + newlength
          + ", replica=" + replica);

      //check replica
      if (replica == null) {
        throw new ReplicaNotFoundException(oldBlock);
      }

      //check replica state
      if (replica.getState() != ReplicaState.RUR) {
        throw new IOException("replica.getState() != " + ReplicaState.RUR
            + ", replica=" + replica);
      }

      //check replica's byte on disk
      if (replica.getBytesOnDisk() != oldBlock.getNumBytes()) {
        throw new IOException("THIS IS NOT SUPPOSED TO HAPPEN:"
            + " replica.getBytesOnDisk() != block.getNumBytes(), block="
            + oldBlock + ", replica=" + replica);
      }

      //check replica files before update
      checkReplicaFiles(replica);

      //update replica
      final ReplicaInfo finalized = updateReplicaUnderRecovery(oldBlock
          .getBlockPoolId(), replica, recoveryId,
          newBlockId, newlength);

      boolean copyTruncate = newBlockId != oldBlock.getBlockId();
      if (!copyTruncate) {
        assert finalized.getBlockId() == oldBlock.getBlockId()
            && finalized.getGenerationStamp() == recoveryId
            && finalized.getNumBytes() == newlength
            : "Replica information mismatched: oldBlock=" + oldBlock

View on GitHub (pinned to 2add963021)

Solutions

Run 'hdfs fsck /path -files -blocks -locations' to find which replica diverged and whether the block is recoverable from other replicas.
Compare the on-disk block file size under <volume>/current/<bpid>/finalized/... with the NameNode's block length from fsck output.
Delete the divergent replica (block file plus its .meta) so the NameNode re-replicates from healthy copies; DirectoryScanner reconciles the volume map.
Check disk health (dmesg, SMART) and evict the bad volume using dfs.datanode.failed.volumes.tolerated.
Upgrade to the latest point release; several block-recovery length races were fixed after HDFS-9187.

Defensive patterns

Strategy: try-catch

Validate before calling

Replica r = fsDataset.getReplica(bpid, block.getBlockId());
if (r != null && r.getBytesOnDisk() != block.getNumBytes()) {
  // replica diverged from NN record; quarantine it instead of recovering
  throw new IOException("replica length mismatch for " + block);
}

Try / catch

try {
  fsDataset.updateReplica(block, recoveryId, newlength);
} catch (IOException e) {
  if (e.getMessage().contains("getBytesOnDisk()")) {
    // treat as corrupt replica: report to NN so it is invalidated and re-replicated
  }
}

Prevention

Enable the DirectoryScanner (dfs.datanode.directoryscan.interval) so volume map and disk stay reconciled.
Monitor disk health and set dfs.datanode.failed.volumes.tolerated to evict bad volumes early.
Schedule periodic 'hdfs fsck' to catch under-replicated or corrupt replicas before recovery needs them.

When it happens

Trigger: Block file truncated or extended on disk by a failing drive or external tool between block report and recovery; a previous recovery committed a different length than what is physically on disk; balancer or scanner replacing the replica mid-recovery; block file restored from backup with a different length.

Common situations: Failing or corrupt disk on one replica; NFS-backed volumes with partial writes; manually restored block directories; recovery length races fixed in later Hadoop point releases.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/43e7e8fc98af25b5. Report an issue: GitHub.