apache/hadoop · error · ReplicaNotFoundException

Cannot append to a replica with unexpected generation stamp

Error message

Cannot append to a replica with unexpected generation stamp {}

What it means

Thrown as ReplicaNotFoundException (UNEXPECTED_GS_REPLICA prefix, 'Cannot append to a replica with unexpected generation stamp') from FsDatasetImpl.getReplicaInfo(ExtendedBlock). The branch is reached when the exact (blockId, generationStamp) lookup misses, but a lookup by blockId alone succeeds: the replica exists, only with a different generation stamp than the caller's ExtendedBlock carries.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:897

  /**
   * Get the meta info of a block stored in volumeMap. To find a block,
   * block pool Id, block Id and generation stamp must match.
   * @param b extended block
   * @return the meta replica information
   * @throws ReplicaNotFoundException if no entry is in the map or 
   *                        there is a generation stamp mismatch
   */
  ReplicaInfo getReplicaInfo(ExtendedBlock b)
      throws ReplicaNotFoundException {
    ReplicaInfo info = volumeMap.get(b.getBlockPoolId(), b.getLocalBlock());
    if (info == null) {
      if (volumeMap.get(b.getBlockPoolId(), b.getLocalBlock().getBlockId())
          == null) {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.NON_EXISTENT_REPLICA + b);
      } else {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + b);
      }
    }
    return info;
  }

  /**
   * Get the meta info of a block stored in volumeMap. Block is looked up
   * without matching the generation stamp.
   * @param bpid block pool Id
   * @param blkid block Id
   * @return the meta replica information; null if block was not found
   * @throws ReplicaNotFoundException if no entry is in the map or 
   *                        there is a generation stamp mismatch
   */
  @VisibleForTesting
  ReplicaInfo getReplicaInfo(String bpid, long blkid)
      throws ReplicaNotFoundException {

View on GitHub (pinned to 2add963021)

Solutions

Re-fetch the block's current generation stamp from the NameNode (getClient().getLocatedBlocks / NAS-returned LocatedBlock) and retry with the fresh ExtendedBlock.
For writers: abandon the stale block via abandonBlock and let the NameNode allocate the correct append target.
Check DataNode logs for 'recoverBlock'/'updateBlock' to confirm the GS bump event that made the caller stale.
In HA setups verify there is a single active NameNode — split-brain recoveries produce exactly this stamp skew.

Defensive patterns

Strategy: validation

Validate before calling

// Compare genstamps before replica-scoped ops.
ReplicaInfo info = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (info == null) {
  return handleMissingReplica(b);
}
if (info.getGenerationStamp() != b.getGenerationStamp()) {
  // caller's stamp is stale: refresh from NameNode
  b = refreshExtendedBlockFromNameNode(dfsClient, b);
}

Try / catch

// Catch and branch on the two ReplicaNotFoundException flavors.
try {
  info = fsDataset.getReplicaInfo(b);
} catch (ReplicaNotFoundException e) {
  if (e.getMessage().contains("unexpected generation stamp")) {
    b = refreshExtendedBlockFromNameNode(dfsClient, b); // retry with fresh GS
    info = fsDataset.getReplicaInfo(b);
  } else {
    throw e; // genuinely missing replica
  }
}

Prevention

After recoverLease/recoverAppend, always use the NameNode-returned block, never the pre-recovery object.
Treat generation stamp as server-authoritative: validate locally before touching the dataset.
In HA clusters, verify a single active NameNode — GS skew is the classic split-brain symptom.

When it happens

Trigger: Append/recover/finalize flows where the client or NameNode passes an ExtendedBlock whose genstamp is older (or newer) than the stored replica's: append after lease recovery bumped the GS, a restarted writer using a pre-recovery GS, or a stale block-sender handshake during pipeline recovery.

Common situations: Client process paused/GC'd across a lease recovery, then resumes appending with its old block object; HA failover causing recovery then duplicate append attempts; block scanner or balancer querying with cached genstamp; test code reusing an ExtendedBlock after recoverAppend.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/28d65e2a344055f8. Report an issue: GitHub.