{"record":{"id":"28d65e2a344055f8","repo":"apache/hadoop","slug":"cannot-append-to-a-replica-with-unexpected-generat","errorCode":null,"errorMessage":"Cannot append to a replica with unexpected generation stamp {}","messagePattern":"Cannot append to a replica with unexpected generation stamp (.+?)","errorType":"exception","errorClass":"ReplicaNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":897,"sourceCode":"\n  /**\n   * Get the meta info of a block stored in volumeMap. To find a block,\n   * block pool Id, block Id and generation stamp must match.\n   * @param b extended block\n   * @return the meta replica information\n   * @throws ReplicaNotFoundException if no entry is in the map or \n   *                        there is a generation stamp mismatch\n   */\n  ReplicaInfo getReplicaInfo(ExtendedBlock b)\n      throws ReplicaNotFoundException {\n    ReplicaInfo info = volumeMap.get(b.getBlockPoolId(), b.getLocalBlock());\n    if (info == null) {\n      if (volumeMap.get(b.getBlockPoolId(), b.getLocalBlock().getBlockId())\n          == null) {\n        throw new ReplicaNotFoundException(\n            ReplicaNotFoundException.NON_EXISTENT_REPLICA + b);\n      } else {\n        throw new ReplicaNotFoundException(\n            ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + b);\n      }\n    }\n    return info;\n  }\n\n  /**\n   * Get the meta info of a block stored in volumeMap. Block is looked up\n   * without matching the generation stamp.\n   * @param bpid block pool Id\n   * @param blkid block Id\n   * @return the meta replica information; null if block was not found\n   * @throws ReplicaNotFoundException if no entry is in the map or \n   *                        there is a generation stamp mismatch\n   */\n  @VisibleForTesting\n  ReplicaInfo getReplicaInfo(String bpid, long blkid)\n      throws ReplicaNotFoundException {","sourceCodeStart":879,"sourceCodeEnd":915,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L879-L915","documentation":"Thrown as ReplicaNotFoundException (UNEXPECTED_GS_REPLICA prefix, 'Cannot append to a replica with unexpected generation stamp') from FsDatasetImpl.getReplicaInfo(ExtendedBlock). The branch is reached when the exact (blockId, generationStamp) lookup misses, but a lookup by blockId alone succeeds: the replica exists, only with a different generation stamp than the caller's ExtendedBlock carries.","triggerScenarios":"Append/recover/finalize flows where the client or NameNode passes an ExtendedBlock whose genstamp is older (or newer) than the stored replica's: append after lease recovery bumped the GS, a restarted writer using a pre-recovery GS, or a stale block-sender handshake during pipeline recovery.","commonSituations":"Client process paused/GC'd across a lease recovery, then resumes appending with its old block object; HA failover causing recovery then duplicate append attempts; block scanner or balancer querying with cached genstamp; test code reusing an ExtendedBlock after recoverAppend.","solutions":["Re-fetch the block's current generation stamp from the NameNode (getClient().getLocatedBlocks / NAS-returned LocatedBlock) and retry with the fresh ExtendedBlock.","For writers: abandon the stale block via abandonBlock and let the NameNode allocate the correct append target.","Check DataNode logs for 'recoverBlock'/'updateBlock' to confirm the GS bump event that made the caller stale.","In HA setups verify there is a single active NameNode — split-brain recoveries produce exactly this stamp skew."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Compare genstamps before replica-scoped ops.\nReplicaInfo info = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (info == null) {\n  return handleMissingReplica(b);\n}\nif (info.getGenerationStamp() != b.getGenerationStamp()) {\n  // caller's stamp is stale: refresh from NameNode\n  b = refreshExtendedBlockFromNameNode(dfsClient, b);\n}","typeGuard":null,"tryCatchPattern":"// Catch and branch on the two ReplicaNotFoundException flavors.\ntry {\n  info = fsDataset.getReplicaInfo(b);\n} catch (ReplicaNotFoundException e) {\n  if (e.getMessage().contains(\"unexpected generation stamp\")) {\n    b = refreshExtendedBlockFromNameNode(dfsClient, b); // retry with fresh GS\n    info = fsDataset.getReplicaInfo(b);\n  } else {\n    throw e; // genuinely missing replica\n  }\n}","preventionTips":["After recoverLease/recoverAppend, always use the NameNode-returned block, never the pre-recovery object.","Treat generation stamp as server-authoritative: validate locally before touching the dataset.","In HA clusters, verify a single active NameNode — GS skew is the classic split-brain symptom."],"tags":["hadoop","hdfs","datanode","fsdataset","generation-stamp","append","lease-recovery"],"backgroundTag":"generation-stamp-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}