apache/hadoop · critical · IOException

rur.getNumBytes() < newlength = {newlength}, rur={rur}

Error message

rur.getNumBytes() < newlength = {newlength}, rur={rur}

What it means

During block recovery the NameNode agrees on a final length (usually the longest replica's). updateReplicaUnderRecovery refuses to set a replica to a length larger than the bytes it physically holds (rur.getNumBytes()); the DataNode never pads a block, because clients would read garbage.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3234

                                          ReplicaInfo rur,
                                          long recoveryId,
                                          long newBlockId,
                                          long newlength) throws IOException {
    //check recovery id
    if (rur.getRecoveryID() != recoveryId) {
      throw new IOException("rur.getRecoveryID() != recoveryId = " + recoveryId
          + ", rur=" + rur);
    }

    boolean copyOnTruncate = newBlockId > 0L && rur.getBlockId() != newBlockId;
    // bump rur's GS to be recovery id
    if(!copyOnTruncate) {
      rur.bumpReplicaGS(recoveryId);
    }

    //update length
    if (rur.getNumBytes() < newlength) {
      throw new IOException("rur.getNumBytes() < newlength = " + newlength
          + ", rur=" + rur);
    }

    if (rur.getNumBytes() > newlength) {
      if(!copyOnTruncate) {
        rur.breakHardLinksIfNeeded();
        rur.truncateBlock(newlength);
        // update RUR with the new length
        rur.setNumBytes(newlength);
      } else {
        // Copying block to a new block with new blockId.
        // Not truncating original block.
        FsVolumeImpl volume = (FsVolumeImpl) rur.getVolume();
        ReplicaInPipeline newReplicaInfo = volume.updateRURCopyOnTruncate(
            rur, bpid, newBlockId, recoveryId, newlength);
        if (newReplicaInfo.getState() != ReplicaState.RBW) {
          throw new IOException("Append on block " + rur.getBlockId()
              + " returned a replica of state " + newReplicaInfo.getState()

View on GitHub (pinned to 2add963021)

Solutions

1. Run 'hdfs fsck' to identify the short replica and confirm healthy copies exist elsewhere.
2. Delete the short replica (block plus meta) on the affected DataNode so the NN re-replicates from a full copy; DirectoryScanner then reconciles.
3. If this DN is short on many blocks, suspect volume corruption: evict the volume, check hardware, and let re-replication rebuild.
4. Upgrade to pick up pipeline-recovery length fixes; if reproducible, capture the recovery RPC log sequence and report.

Replica r = fsDataset.getReplica(bpid, block.getBlockId());
if (r != null && r.getNumBytes() < newlength) {
  // this replica is short; exclude it from the length agreement and mark corrupt
}

try {
  fsDataset.updateReplica(block, recoveryId, newlength);
} catch (IOException e) {
  if (e.getMessage().contains("getNumBytes() < newlength")) {
    // replica cannot be padded: invalidate it and let NN re-replicate
  }
}

When it happens

Trigger: Pipeline write where this DataNode died before receiving the tail that other replicas received; recovery length chosen from a longer replica; block file shortened by disk failure or external truncation; recovery racing an in-progress flush so this replica is behind.

Common situations: DN crash or network partition mid-write followed by lease recovery; slow or failing disk losing the last chunk; cluster instability leaving mixed replica states.

Related errors

• THIS IS NOT SUPPOSED TO HAPPEN: replica.getBytesOnDisk() !=
• Replica was found but missing fields.
• {} has no enough internal blocks(current: {}), unable to sta
• Replica gen stamp < block genstamp, block={block}, replica={
• Checksum failed at {failedPos} for replica: {replica}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c0dc98081ad9f7a3. Report an issue: GitHub.