apache/hadoop · error · IOException

replica.getGenerationStamp() < block.getGenerationStamp(), b

Error message

replica.getGenerationStamp() < block.getGenerationStamp(), block={block}, replica={replica}

What it means

IOException thrown by FsDatasetImpl.initReplicaRecoveryImpl when the local replica's generation stamp is strictly older than the generation stamp of the RecoveringBlock the NameNode dispatched (replica.getGenerationStamp() < block.getGenerationStamp()). A stale replica cannot participate in recovery that would produce the newer GS, so the DataNode rejects it up front.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3100

        replica.getState() == ReplicaState.RBW) {
      final ReplicaInPipeline rip = (ReplicaInPipeline)replica;
      if (!rip.attemptToSetWriter(null, Thread.currentThread())) {
        throw new MustStopExistingWriter(rip);
      }

      //check replica bytes on disk.
      if (replica.getBytesOnDisk() < replica.getVisibleLength()) {
        throw new IOException("getBytesOnDisk() < getVisibleLength(), rip="
            + replica);
      }

      //check the replica's files
      checkReplicaFiles(replica);
    }

    //check generation stamp
    if (replica.getGenerationStamp() < block.getGenerationStamp()) {
      throw new IOException(
          "replica.getGenerationStamp() < block.getGenerationStamp(), block="
          + block + ", replica=" + replica);
    }

    //check recovery id
    if (replica.getGenerationStamp() >= recoveryId) {
      throw new IOException("THIS IS NOT SUPPOSED TO HAPPEN:"
          + " replica.getGenerationStamp() >= recoveryId = " + recoveryId
          + ", block=" + block + ", replica=" + replica);
    }

    //check RUR
    final ReplicaInfo rur;
    if (replica.getState() == ReplicaState.RUR) {
      rur = replica;
      if (rur.getRecoveryID() >= recoveryId) {
        throw new RecoveryInProgressException(
            "rur.getRecoveryID() >= recoveryId = " + recoveryId

View on GitHub (pinned to 2add963021)

Solutions

1. Usually self-healing: the NameNode discards the stale replica at the next block report; no action needed for one-off occurrences.
2. Force a block report (hdfs dfsadmin -triggerBlockReport -datanode <dn>) to refresh NameNode state quickly.
3. Run hdfs fsck to confirm the block has replicas at the current GS.
4. If it recurs on the same DataNode, inspect why block reports are delayed (RPC queue, NN overload).

boolean isStaleGenerationStamp(IOException e) {
  return e.getMessage() != null
      && e.getMessage().startsWith("replica.getGenerationStamp() < block.getGenerationStamp()");
}

try {
  dataset.initReplicaRecovery(rBlock);
} catch (IOException e) {
  if (e.getMessage() != null
      && e.getMessage().contains("replica.getGenerationStamp() < block.getGenerationStamp()")) {
    // stale replica: skip; re-run recovery excludes it after next block report
    return recoverFromOtherReplicas(rBlock);
  }
  throw e;
}

When it happens

Trigger: initReplicaRecovery(block, recoveryId) where the block carried in the recovery request already has a higher GS than this DataNode's copy - e.g., a previous lease recovery bumped the GS via another DataNode, and the NameNode still lists this stale replica for a new recovery attempt.

Common situations: Stale replica lingering after lease recovery completed elsewhere (NameNode block report not yet processed); block re-opened via lease steal with GS bump; asynchronous block report lag after invalidate.

Related errors

• Replica gen stamp < block genstamp, block={block}, replica={
• ProvidedReplica does not yet support writes
• The new recovery id: {} must be greater than the current one
• Cannot append to a replica with unexpected generation stamp
• Cannot append to a replica with unexpected generation stamp

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b457b1adfc9e7507. Report an issue: GitHub.