{"record":{"id":"05973fc8d2bd9db2","repo":"apache/hadoop","slug":"getbytesondisk-getvisiblelength-rip-replic","errorCode":null,"errorMessage":"getBytesOnDisk() < getVisibleLength(), rip={replica}","messagePattern":"getBytesOnDisk\\(\\) < getVisibleLength\\(\\), rip=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3090,"sourceCode":"      Block block, long recoveryId)\n          throws IOException, MustStopExistingWriter {\n    final ReplicaInfo replica = map.get(bpid, block.getBlockId());\n    //check replica\n    if (replica == null) {\n      return null;\n    }\n\n    //stop writer if there is any\n    if (replica.getState() == ReplicaState.TEMPORARY ||\n        replica.getState() == ReplicaState.RBW) {\n      final ReplicaInPipeline rip = (ReplicaInPipeline)replica;\n      if (!rip.attemptToSetWriter(null, Thread.currentThread())) {\n        throw new MustStopExistingWriter(rip);\n      }\n\n      //check replica bytes on disk.\n      if (replica.getBytesOnDisk() < replica.getVisibleLength()) {\n        throw new IOException(\"getBytesOnDisk() < getVisibleLength(), rip=\"\n            + replica);\n      }\n\n      //check the replica's files\n      checkReplicaFiles(replica);\n    }\n\n    //check generation stamp\n    if (replica.getGenerationStamp() < block.getGenerationStamp()) {\n      throw new IOException(\n          \"replica.getGenerationStamp() < block.getGenerationStamp(), block=\"\n          + block + \", replica=\" + replica);\n    }\n\n    //check recovery id\n    if (replica.getGenerationStamp() >= recoveryId) {\n      throw new IOException(\"THIS IS NOT SUPPOSED TO HAPPEN:\"\n          + \" replica.getGenerationStamp() >= recoveryId = \" + recoveryId","sourceCodeStart":3072,"sourceCodeEnd":3108,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3072-L3108","documentation":"IOException thrown by FsDatasetImpl.initReplicaRecoveryImpl when a TEMPORARY/RBW replica being enlisted for block recovery has fewer bytes on disk than its visible length (getBytesOnDisk() < getVisibleLength()). Recovery would finalize a length clients can already see but that is not physically on disk, so the DataNode refuses.","triggerScenarios":"initReplicaRecovery on a replica in pipeline state where the visible length (acked to the NameNode through hflush) exceeds what was actually persisted to the block file - e.g., acked-in-memory bytes lost before fsync on a crash, or a truncated file after disk error.","commonSituations":"DataNode crash between packet ack and disk flush; replica visible length advanced by hflush while disk write lagged; failing disk silently truncating writes; replica resurrected from an inconsistent snapshot.","solutions":["Recovery will proceed on other replicas: verify with hdfs fsck that at least one pipeline member is healthy.","Investigate DataNode logs around the write window for disk errors or 'Slow flush' warnings.","Test the disk (smartctl) and replace it if truncation is hardware-caused.","If all replicas show this, salvage with hdfs debug recoverLease and accept the visible-length data may be unrecoverable."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"boolean isTruncatedReplica(IOException e) {\n  return e.getMessage() != null && e.getMessage().startsWith(\"getBytesOnDisk() < getVisibleLength()\");\n}","tryCatchPattern":"try {\n  ReplicaRecoveryInfo info = dataset.initReplicaRecovery(rBlock);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"getBytesOnDisk() < getVisibleLength()\")) {\n    // this replica cannot honor acked length: recover from other pipeline members\n    return recoverFromOtherReplicas(rBlock);\n  }\n  throw e;\n}","preventionTips":["Prefer replicas with durable acks: keep hflush discipline so visible length tracks fsynced bytes.","Investigate any DataNode reporting this - acked-but-not-durable points at disk/GC stalls.","Test disks after crashes; silent truncation is a hardware trust problem."],"tags":["hdfs","datanode","block-recovery","truncated-block","visible-length"],"backgroundTag":"block-file-truncated","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}