{"record":{"id":"ce5f0cca6def1d80","repo":"apache/hadoop","slug":"cannot-append-to-a-replica-with-unexpected-generat-ce5f0c","errorCode":null,"errorMessage":"Cannot append to a replica with unexpected generation stamp {b}. Expected GS range is [{generationStamp}, {newGS}].","messagePattern":"Cannot append to a replica with unexpected generation stamp (.+?)\\. Expected GS range is \\[(.+?), (.+?)\\]\\.","errorType":"exception","errorClass":"ReplicaNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1749,"sourceCode":"    } finally {\n      if (dataNodeMetrics != null) {\n        long recoverRbwMs = Time.monotonicNow() - startTimeMs;\n        dataNodeMetrics.addRecoverRbwOp(recoverRbwMs);\n      }\n    }\n  }\n\n  private ReplicaHandler recoverRbwImpl(ReplicaInPipeline rbw,\n      ExtendedBlock b, long newGS, long minBytesRcvd, long maxBytesRcvd)\n      throws IOException {\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n        b.getBlockPoolId(), getStorageUuidForLock(b),\n        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n      // check generation stamp\n      long replicaGenerationStamp = rbw.getGenerationStamp();\n      if (replicaGenerationStamp < b.getGenerationStamp() ||\n          replicaGenerationStamp > newGS) {\n        throw new ReplicaNotFoundException(\n            ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + b +\n                \". Expected GS range is [\" + b.getGenerationStamp() + \", \" +\n                newGS + \"].\");\n      }\n\n      // check replica length\n      long bytesAcked = rbw.getBytesAcked();\n      long numBytes = rbw.getNumBytes();\n      if (bytesAcked < minBytesRcvd || numBytes > maxBytesRcvd) {\n        throw new ReplicaNotFoundException(\"Unmatched length replica \" +\n            rbw + \": BytesAcked = \" + bytesAcked +\n            \" BytesRcvd = \" + numBytes + \" are not in the range of [\" +\n            minBytesRcvd + \", \" + maxBytesRcvd + \"].\");\n      }\n\n      long bytesOnDisk = rbw.getBytesOnDisk();\n      long blockDataLength = rbw.getReplicaInfo().getBlockDataLength();\n      if (bytesOnDisk != blockDataLength) {","sourceCodeStart":1731,"sourceCodeEnd":1767,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1731-L1767","documentation":"Same generation-stamp range contract as recoverCheck but for the RBW fast path: recoverRbwImpl requires the RBW replica's GS to lie in [b.getGenerationStamp(), newGS]. Outside that range the DataNode refuses with ReplicaNotFoundException (UNEXPECTED_GS_REPLICA), the message embedding the block b with its stamp.","triggerScenarios":"Pipeline recovery where this DN's RBW replica carries a GS older than the block's (stale from a previous incarnation) or newer than newGS (a different recovery already advanced it).","commonSituations":"Client rebuilding a pipeline from stale located blocks after another recovery bumped GSs; concurrent lease and block recovery interleaving; retry storms reusing outdated ExtendedBlocks.","solutions":["Refresh located blocks and retry with the current GS range","Verify the block GS on the NN vs DN logs; a replica persistently above newGS should be invalidated","Let one recovery finish before starting the next (lease serialization enforces this)","Run 'hdfs fsck' to reconcile block and replica GS state if it loops"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nlong gs = (r != null) ? r.getGenerationStamp() : -1L;\nif (gs < b.getGenerationStamp() || gs > newGS) {\n  refreshLocatedBlocksAndRebuildPipeline(); // GSs are stale somewhere upstream\n  return;\n}\nfsDataset.recoverRbw(b, newGS, minBytesRcvd, maxBytesRcvd);","typeGuard":null,"tryCatchPattern":"catch (ReplicaNotFoundException rnfe) {\n  if (rnfe.getMessage().contains(ReplicaNotFoundException.UNEXPECTED_GS_REPLICA)) {\n    refreshLocatedBlocksAndRetryRecoveryOnce();\n  } else { throw rnfe; }\n}","preventionTips":["Pair the ExtendedBlock GS with the targeted replica; never mix stamps across retries","Re-fetch located blocks after any lease-recovery event before rebuilding a pipeline","Serialize recoveries per block so two incarnations never race"],"tags":["hdfs","datanode","block-recovery","rbw","generation-stamp"],"backgroundTag":"hdfs-generation-stamp-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}