{"record":{"id":"d9e02364f7920f9b","repo":"apache/hadoop","slug":"replica-gen-stamp-block-genstamp-block-block","errorCode":null,"errorMessage":"Replica gen stamp < block genstamp, block={block}, replica={replica}","messagePattern":"Replica gen stamp < block genstamp, block=(.+?), replica=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java","lineNumber":275,"sourceCode":"      ChunkChecksum chunkChecksum = null;\n      final long replicaVisibleLength;\n      try (AutoCloseableLock lock = datanode.getDataSetLockManager().readLock(\n          LockLevel.BLOCK_POOl, block.getBlockPoolId())) {\n        replica = getReplica(block, datanode);\n        replicaVisibleLength = replica.getVisibleLength();\n      }\n      if (replica.getState() == ReplicaState.RBW) {\n        final ReplicaInPipeline rbw = (ReplicaInPipeline) replica;\n        rbw.waitForMinLength(startOffset + length, 3, TimeUnit.SECONDS);\n        chunkChecksum = rbw.getLastChecksumAndDataLen();\n      }\n      if (replica instanceof FinalizedReplica) {\n        chunkChecksum = getPartialChunkChecksumForFinalized(\n            (FinalizedReplica)replica);\n      }\n\n      if (replica.getGenerationStamp() < block.getGenerationStamp()) {\n        throw new IOException(\"Replica gen stamp < block genstamp, block=\"\n            + block + \", replica=\" + replica);\n      } else if (replica.getGenerationStamp() > block.getGenerationStamp()) {\n        if (DataNode.LOG.isDebugEnabled()) {\n          DataNode.LOG.debug(\"Bumping up the client provided\"\n              + \" block's genstamp to latest \" + replica.getGenerationStamp()\n              + \" for block \" + block);\n        }\n        block.setGenerationStamp(replica.getGenerationStamp());\n      }\n      if (replicaVisibleLength < 0) {\n        throw new IOException(\"Replica is not readable, block=\"\n            + block + \", replica=\" + replica);\n      }\n      if (DataNode.LOG.isDebugEnabled()) {\n        DataNode.LOG.debug(\"block=\" + block + \", replica=\" + replica);\n      }\n\n      // transferToFully() fails on 32 bit platforms for block sizes >= 2GB,","sourceCodeStart":257,"sourceCodeEnd":293,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java#L257-L293","documentation":"While opening a BlockSender for a read, the local replica's generation stamp must not be older than the generation stamp carried in the incoming ExtendedBlock. A replica GS lower than the block GS means this datanode's copy predates a block recovery/append that bumped the GS, so the bytes on disk do not correspond to the identity the client asked for; reading them would serve the wrong block generation, hence the IOException.","triggerScenarios":"A readBlock request arrives whose ExtendedBlock.getGenerationStamp() is newer than replica.getGenerationStamp(): typically the client's LocatedBlock was refreshed after lease recovery bumped the GS on other datanodes, but this DN's replica was finalized with the old GS (or its updateBlock failed during recovery — see the 2200 path).","commonSituations":"Client cached block locations across a lease-recovery event; a datanode that failed the updateBlock step of recovery and now serves a stale replica; races between an append/recovery and a concurrent read.","solutions":["Let the client retry: DFSClient refreshes located blocks from the NameNode and re-reads from a healthy replica, which is the normal self-heal path.","On the datanode, confirm the replica recovered correctly (no 'Failed to updateBlock' WARN for this block) and that its GS matches what the NN reports via fsck.","If the replica is permanently stale, delete the bad replica (or its directory entry) so the NN re-replicates a current copy.","For recurring races, avoid long-cached LocatedBlock data in custom clients — always re-fetch locations on GenstampMismatchException-style failures."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Client-side: compare GS in located block vs a fresh fetch before reading\nLocatedBlock fresh = namenode.getBlockLocations(file, offset, 1).get(0);\nif (fresh.getBlock().getGenerationStamp() != cached.getBlock().getGenerationStamp()) {\n  cached = fresh; // use current GS so the DN replica comparison passes\n}","typeGuard":null,"tryCatchPattern":"try {\n  reader = buildBlockReader(cachedLocatedBlock);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Replica gen stamp < block genstamp\")) {\n    cachedLocatedBlock = namenode.getBlockLocations(file, offset, len).get(0);\n    reader = buildBlockReader(cachedLocatedBlock); // retry with refreshed GS\n  } else {\n    throw e;\n  }\n}","preventionTips":["Never cache LocatedBlock/ExtendedBlock across lease-recovery or append events — refetch on read failures.","Distribute custom readers over multiple replicas so a single stale replica is a retry, not an outage.","Watch DN logs for 'Failed to updateBlock' — a replica left with an old GS is the usual seed for this error."],"tags":["hdfs","datanode","read-path","generation-stamp","block-recovery"],"backgroundTag":"generation-stamp-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}