{"record":{"id":"b457b1adfc9e7507","repo":"apache/hadoop","slug":"replica-getgenerationstamp-block-getgeneration","errorCode":null,"errorMessage":"replica.getGenerationStamp() < block.getGenerationStamp(), block={block}, replica={replica}","messagePattern":"replica\\.getGenerationStamp\\(\\) < block\\.getGenerationStamp\\(\\), block=(.+?), replica=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":3100,"sourceCode":"        replica.getState() == ReplicaState.RBW) {\n      final ReplicaInPipeline rip = (ReplicaInPipeline)replica;\n      if (!rip.attemptToSetWriter(null, Thread.currentThread())) {\n        throw new MustStopExistingWriter(rip);\n      }\n\n      //check replica bytes on disk.\n      if (replica.getBytesOnDisk() < replica.getVisibleLength()) {\n        throw new IOException(\"getBytesOnDisk() < getVisibleLength(), rip=\"\n            + replica);\n      }\n\n      //check the replica's files\n      checkReplicaFiles(replica);\n    }\n\n    //check generation stamp\n    if (replica.getGenerationStamp() < block.getGenerationStamp()) {\n      throw new IOException(\n          \"replica.getGenerationStamp() < block.getGenerationStamp(), block=\"\n          + block + \", replica=\" + replica);\n    }\n\n    //check recovery id\n    if (replica.getGenerationStamp() >= recoveryId) {\n      throw new IOException(\"THIS IS NOT SUPPOSED TO HAPPEN:\"\n          + \" replica.getGenerationStamp() >= recoveryId = \" + recoveryId\n          + \", block=\" + block + \", replica=\" + replica);\n    }\n\n    //check RUR\n    final ReplicaInfo rur;\n    if (replica.getState() == ReplicaState.RUR) {\n      rur = replica;\n      if (rur.getRecoveryID() >= recoveryId) {\n        throw new RecoveryInProgressException(\n            \"rur.getRecoveryID() >= recoveryId = \" + recoveryId","sourceCodeStart":3082,"sourceCodeEnd":3118,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L3082-L3118","documentation":"IOException thrown by FsDatasetImpl.initReplicaRecoveryImpl when the local replica's generation stamp is strictly older than the generation stamp of the RecoveringBlock the NameNode dispatched (replica.getGenerationStamp() < block.getGenerationStamp()). A stale replica cannot participate in recovery that would produce the newer GS, so the DataNode rejects it up front.","triggerScenarios":"initReplicaRecovery(block, recoveryId) where the block carried in the recovery request already has a higher GS than this DataNode's copy - e.g., a previous lease recovery bumped the GS via another DataNode, and the NameNode still lists this stale replica for a new recovery attempt.","commonSituations":"Stale replica lingering after lease recovery completed elsewhere (NameNode block report not yet processed); block re-opened via lease steal with GS bump; asynchronous block report lag after invalidate.","solutions":["Usually self-healing: the NameNode discards the stale replica at the next block report; no action needed for one-off occurrences.","Force a block report (hdfs dfsadmin -triggerBlockReport -datanode <dn>) to refresh NameNode state quickly.","Run hdfs fsck to confirm the block has replicas at the current GS.","If it recurs on the same DataNode, inspect why block reports are delayed (RPC queue, NN overload)."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":"boolean isStaleGenerationStamp(IOException e) {\n  return e.getMessage() != null\n      && e.getMessage().startsWith(\"replica.getGenerationStamp() < block.getGenerationStamp()\");\n}","tryCatchPattern":"try {\n  dataset.initReplicaRecovery(rBlock);\n} catch (IOException e) {\n  if (e.getMessage() != null\n      && e.getMessage().contains(\"replica.getGenerationStamp() < block.getGenerationStamp()\")) {\n    // stale replica: skip; re-run recovery excludes it after next block report\n    return recoverFromOtherReplicas(rBlock);\n  }\n  throw e;\n}","preventionTips":["Trigger a block report (hdfs dfsadmin -triggerBlockReport) after recovery storms to purge stale entries.","Keep NameNode block-processing queues healthy so reports are not delayed.","Expect stale-replica noise around lease recoveries; only act if fsck shows under-replication."],"tags":["hdfs","datanode","block-recovery","generation-stamp","stale-replica"],"backgroundTag":"generation-stamp-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}