{"record":{"id":"c0dc98081ad9f7a3","repo":"apache/hadoop","slug":"rur-getnumbytes-newlength-newlength-rur","errorCode":null,"errorMessage":"rur.getNumBytes() < newlength = {newlength}, rur={rur}","messagePattern":"rur\\.getNumBytes\\(\\) < newlength = (.+?), rur=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3234,"sourceCode":"                                          ReplicaInfo rur,\n                                          long recoveryId,\n                                          long newBlockId,\n                                          long newlength) throws IOException {\n    //check recovery id\n    if (rur.getRecoveryID() != recoveryId) {\n      throw new IOException(\"rur.getRecoveryID() != recoveryId = \" + recoveryId\n          + \", rur=\" + rur);\n    }\n\n    boolean copyOnTruncate = newBlockId > 0L && rur.getBlockId() != newBlockId;\n    // bump rur's GS to be recovery id\n    if(!copyOnTruncate) {\n      rur.bumpReplicaGS(recoveryId);\n    }\n\n    //update length\n    if (rur.getNumBytes() < newlength) {\n      throw new IOException(\"rur.getNumBytes() < newlength = \" + newlength\n          + \", rur=\" + rur);\n    }\n\n    if (rur.getNumBytes() > newlength) {\n      if(!copyOnTruncate) {\n        rur.breakHardLinksIfNeeded();\n        rur.truncateBlock(newlength);\n        // update RUR with the new length\n        rur.setNumBytes(newlength);\n      } else {\n        // Copying block to a new block with new blockId.\n        // Not truncating original block.\n        FsVolumeImpl volume = (FsVolumeImpl) rur.getVolume();\n        ReplicaInPipeline newReplicaInfo = volume.updateRURCopyOnTruncate(\n            rur, bpid, newBlockId, recoveryId, newlength);\n        if (newReplicaInfo.getState() != ReplicaState.RBW) {\n          throw new IOException(\"Append on block \" + rur.getBlockId()\n              + \" returned a replica of state \" + newReplicaInfo.getState()","sourceCodeStart":3216,"sourceCodeEnd":3252,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3216-L3252","documentation":"During block recovery the NameNode agrees on a final length (usually the longest replica's). updateReplicaUnderRecovery refuses to set a replica to a length larger than the bytes it physically holds (rur.getNumBytes()); the DataNode never pads a block, because clients would read garbage.","triggerScenarios":"Pipeline write where this DataNode died before receiving the tail that other replicas received; recovery length chosen from a longer replica; block file shortened by disk failure or external truncation; recovery racing an in-progress flush so this replica is behind.","commonSituations":"DN crash or network partition mid-write followed by lease recovery; slow or failing disk losing the last chunk; cluster instability leaving mixed replica states.","solutions":["Run 'hdfs fsck' to identify the short replica and confirm healthy copies exist elsewhere.","Delete the short replica (block plus meta) on the affected DataNode so the NN re-replicates from a full copy; DirectoryScanner then reconciles.","If this DN is short on many blocks, suspect volume corruption: evict the volume, check hardware, and let re-replication rebuild.","Upgrade to pick up pipeline-recovery length fixes; if reproducible, capture the recovery RPC log sequence and report."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Replica r = fsDataset.getReplica(bpid, block.getBlockId());\nif (r != null && r.getNumBytes() < newlength) {\n  // this replica is short; exclude it from the length agreement and mark corrupt\n}","typeGuard":null,"tryCatchPattern":"try {\n  fsDataset.updateReplica(block, recoveryId, newlength);\n} catch (IOException e) {\n  if (e.getMessage().contains(\"getNumBytes() < newlength\")) {\n    // replica cannot be padded: invalidate it and let NN re-replicate\n  }\n}","preventionTips":["Run 'hdfs fsck' regularly to catch short replicas before they block recovery.","Watch for DN disk failures; short block files usually trace to failing hardware.","Avoid kill -9 on DataNodes during heavy pipeline writes."],"tags":["hdfs","datanode","block-recovery","data-corruption","pipeline"],"backgroundTag":"hdfs-replica-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}