{"record":{"id":"3563e9b1a0280053","repo":"apache/hadoop","slug":"rbw-replica-replicainfo-bytesrcvd-numbytes-by","errorCode":null,"errorMessage":"RBW replica {replicaInfo}bytesRcvd({numBytes}), bytesOnDisk({bytesOnDisk}), and bytesAcked({bytesAcked}) are not the same.","messagePattern":"RBW replica (.+?)bytesRcvd\\((.+?)\\), bytesOnDisk\\((.+?)\\), and bytesAcked\\((.+?)\\) are not the same\\.","errorType":"exception","errorClass":"ReplicaAlreadyExistsException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1531,"sourceCode":"    if (replicaGenerationStamp < b.getGenerationStamp() ||\n        replicaGenerationStamp > newGS) {\n      throw new ReplicaNotFoundException(\n          ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + replicaGenerationStamp\n          + \". Expected GS range is [\" + b.getGenerationStamp() + \", \" + \n          newGS + \"].\");\n    }\n    \n    // stop the previous writer before check a replica's length\n    long replicaLen = replicaInfo.getNumBytes();\n    if (replicaInfo.getState() == ReplicaState.RBW) {\n      ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;\n      if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {\n        throw new MustStopExistingWriter(rbw);\n      }\n      // check length: bytesRcvd, bytesOnDisk, and bytesAcked should be the same\n      if (replicaLen != rbw.getBytesOnDisk() \n          || replicaLen != rbw.getBytesAcked()) {\n        throw new ReplicaAlreadyExistsException(\"RBW replica \" + replicaInfo + \n            \"bytesRcvd(\" + rbw.getNumBytes() + \"), bytesOnDisk(\" + \n            rbw.getBytesOnDisk() + \"), and bytesAcked(\" + rbw.getBytesAcked() +\n            \") are not the same.\");\n      }\n    }\n    \n    // check block length\n    if (replicaLen != expectedBlockLen) {\n      throw new IOException(\"Corrupted replica \" + replicaInfo + \n          \" with a length of \" + replicaLen + \n          \" expected length is \" + expectedBlockLen);\n    }\n    \n    return replicaInfo;\n  }\n\n  @Override  // FsDatasetSpi\n  public ReplicaHandler recoverAppend(","sourceCodeStart":1513,"sourceCodeEnd":1549,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1513-L1549","documentation":"Before recovering an RBW replica, recoverCheck requires its three length counters to agree: bytesRcvd (getNumBytes), bytesOnDisk and bytesAcked must all equal replicaLen. If they differ, the replica holds an unacknowledged or unflushed tail and cannot be recovered as-is, so ReplicaAlreadyExistsException is thrown.","triggerScenarios":"recoverAppend/recoverClose touching an RBW replica where data was received but not flushed (numBytes > bytesOnDisk) or not yet acked upstream (numBytes > bytesAcked) - the typical aftermath of a writer crashing mid-packet, sampled at the wrong moment.","commonSituations":"Lease recovery racing a still-active writer; DataNode killed mid-packet; packet-ack lag making the counters transiently unequal exactly when recovery samples them.","solutions":["Retry the recovery - once the writer is stopped and the tail truncated, the counters converge and the next attempt succeeds (the NN retries block recovery)","Verify the previous writer is really dead (no orphaned DataXceiver threads); take a DN thread dump if the error repeats","If permanently stuck, remove the RBW replica: the NN re-replicates from the committed length","Close/abort application streams deterministically so recovery never meets a half-written tail"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"ReplicaInfo raw = (ReplicaInfo) fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (raw != null && raw.getState() == ReplicaState.RBW) {\n  ReplicaInPipeline rbw = (ReplicaInPipeline) raw;\n  if (rbw.getNumBytes() != rbw.getBytesOnDisk()\n      || rbw.getNumBytes() != rbw.getBytesAcked()) {\n    stopWriterAndWaitForQuiesce(rbw); // counters converge, then recover\n    return;\n  }\n}\nfsDataset.recoverAppend(b, newGS, expectedBlockLen);","typeGuard":"boolean isQuiescentRbw(ReplicaInPipeline rbw) {\n  return rbw.getState() == ReplicaState.RBW\n      && rbw.getNumBytes() == rbw.getBytesOnDisk()\n      && rbw.getNumBytes() == rbw.getBytesAcked();\n}","tryCatchPattern":"catch (ReplicaAlreadyExistsException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"RBW replica\")) {\n    stopPreviousWriterThenRetryRecoveryOnce(); // tail truncation converges counters\n  } else { throw e; }\n}","preventionTips":["Abort or close client streams cleanly so RBW replicas are truncated/finalized deterministically","Before invoking recovery, ensure no writer thread still owns the replica (attemptToSetWriter semantics)","Retry recovery after stopping the previous writer instead of failing hard on first counter mismatch"],"tags":["hdfs","datanode","block-recovery","rbw","length-mismatch"],"backgroundTag":"hdfs-replica-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}