apache/hadoop · error · IOException

Replica gen stamp < block genstamp, block={block}, replica={

Error message

Replica gen stamp < block genstamp, block={block}, replica={replica}

What it means

While opening a BlockSender for a read, the local replica's generation stamp must not be older than the generation stamp carried in the incoming ExtendedBlock. A replica GS lower than the block GS means this datanode's copy predates a block recovery/append that bumped the GS, so the bytes on disk do not correspond to the identity the client asked for; reading them would serve the wrong block generation, hence the IOException.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java:275

      ChunkChecksum chunkChecksum = null;
      final long replicaVisibleLength;
      try (AutoCloseableLock lock = datanode.getDataSetLockManager().readLock(
          LockLevel.BLOCK_POOl, block.getBlockPoolId())) {
        replica = getReplica(block, datanode);
        replicaVisibleLength = replica.getVisibleLength();
      }
      if (replica.getState() == ReplicaState.RBW) {
        final ReplicaInPipeline rbw = (ReplicaInPipeline) replica;
        rbw.waitForMinLength(startOffset + length, 3, TimeUnit.SECONDS);
        chunkChecksum = rbw.getLastChecksumAndDataLen();
      }
      if (replica instanceof FinalizedReplica) {
        chunkChecksum = getPartialChunkChecksumForFinalized(
            (FinalizedReplica)replica);
      }

      if (replica.getGenerationStamp() < block.getGenerationStamp()) {
        throw new IOException("Replica gen stamp < block genstamp, block="
            + block + ", replica=" + replica);
      } else if (replica.getGenerationStamp() > block.getGenerationStamp()) {
        if (DataNode.LOG.isDebugEnabled()) {
          DataNode.LOG.debug("Bumping up the client provided"
              + " block's genstamp to latest " + replica.getGenerationStamp()
              + " for block " + block);
        }
        block.setGenerationStamp(replica.getGenerationStamp());
      }
      if (replicaVisibleLength < 0) {
        throw new IOException("Replica is not readable, block="
            + block + ", replica=" + replica);
      }
      if (DataNode.LOG.isDebugEnabled()) {
        DataNode.LOG.debug("block=" + block + ", replica=" + replica);
      }

      // transferToFully() fails on 32 bit platforms for block sizes >= 2GB,

View on GitHub (pinned to 2add963021)

Solutions

1. Let the client retry: DFSClient refreshes located blocks from the NameNode and re-reads from a healthy replica, which is the normal self-heal path.
2. On the datanode, confirm the replica recovered correctly (no 'Failed to updateBlock' WARN for this block) and that its GS matches what the NN reports via fsck.
3. If the replica is permanently stale, delete the bad replica (or its directory entry) so the NN re-replicates a current copy.
4. For recurring races, avoid long-cached LocatedBlock data in custom clients — always re-fetch locations on GenstampMismatchException-style failures.

// Client-side: compare GS in located block vs a fresh fetch before reading
LocatedBlock fresh = namenode.getBlockLocations(file, offset, 1).get(0);
if (fresh.getBlock().getGenerationStamp() != cached.getBlock().getGenerationStamp()) {
  cached = fresh; // use current GS so the DN replica comparison passes
}

try {
  reader = buildBlockReader(cachedLocatedBlock);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Replica gen stamp < block genstamp")) {
    cachedLocatedBlock = namenode.getBlockLocations(file, offset, len).get(0);
    reader = buildBlockReader(cachedLocatedBlock); // retry with refreshed GS
  } else {
    throw e;
  }
}

When it happens

Trigger: A readBlock request arrives whose ExtendedBlock.getGenerationStamp() is newer than replica.getGenerationStamp(): typically the client's LocatedBlock was refreshed after lease recovery bumped the GS on other datanodes, but this DN's replica was finalized with the old GS (or its updateBlock failed during recovery — see the 2200 path).

Common situations: Client cached block locations across a lease-recovery event; a datanode that failed the updateBlock step of recovery and now serves a stale replica; races between an append/recovery and a concurrent read.

Related errors

• ProvidedReplica does not yet support writes
• The new recovery id: {} must be greater than the current one
• Cannot append to a replica with unexpected generation stamp
• Cannot append to a replica with unexpected generation stamp
• replica.getGenerationStamp() < block.getGenerationStamp(), b

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d9e02364f7920f9b. Report an issue: GitHub.