apache/hadoop · error · ReplicaNotFoundException

Cannot append to a replica with unexpected generation stamp

Error message

Cannot append to a replica with unexpected generation stamp {replicaGenerationStamp}. Expected GS range is [{generationStamp}, {newGS}].

What it means

In recoverCheck, the local replica's generation stamp must lie in the closed range [b.getGenerationStamp(), newGS]: not older than the stamp the client/NN knows, and not newer than the recovery target stamp. Outside that range the DataNode throws ReplicaNotFoundException with the UNEXPECTED_GS_REPLICA prefix, flagging a stale (older) or divergent (newer) replica.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1515

    }
  }

  private ReplicaInfo recoverCheck(ExtendedBlock b, long newGS, 
      long expectedBlockLen) throws IOException, MustStopExistingWriter {
    ReplicaInfo replicaInfo = getReplicaInfo(b.getBlockPoolId(), b.getBlockId());
    
    // check state
    if (replicaInfo.getState() != ReplicaState.FINALIZED &&
        replicaInfo.getState() != ReplicaState.RBW) {
      throw new ReplicaNotFoundException(
          ReplicaNotFoundException.UNFINALIZED_AND_NONRBW_REPLICA + replicaInfo);
    }

    // check generation stamp
    long replicaGenerationStamp = replicaInfo.getGenerationStamp();
    if (replicaGenerationStamp < b.getGenerationStamp() ||
        replicaGenerationStamp > newGS) {
      throw new ReplicaNotFoundException(
          ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + replicaGenerationStamp
          + ". Expected GS range is [" + b.getGenerationStamp() + ", " + 
          newGS + "].");
    }
    
    // stop the previous writer before check a replica's length
    long replicaLen = replicaInfo.getNumBytes();
    if (replicaInfo.getState() == ReplicaState.RBW) {
      ReplicaInPipeline rbw = (ReplicaInPipeline) replicaInfo;
      if (!rbw.attemptToSetWriter(null, Thread.currentThread())) {
        throw new MustStopExistingWriter(rbw);
      }
      // check length: bytesRcvd, bytesOnDisk, and bytesAcked should be the same
      if (replicaLen != rbw.getBytesOnDisk() 
          || replicaLen != rbw.getBytesAcked()) {
        throw new ReplicaAlreadyExistsException("RBW replica " + replicaInfo + 
            "bytesRcvd(" + rbw.getNumBytes() + "), bytesOnDisk(" + 
            rbw.getBytesOnDisk() + "), and bytesAcked(" + rbw.getBytesAcked() +

View on GitHub (pinned to 2add963021)

Solutions

1. Refresh block locations on the client and retry - the rebuilt pipeline carries the current GS range
2. Verify the block GS on the NN ('hdfs fsck -files -blocks -locations') against the replica GS in DN logs
3. If one DN replica persistently has a GS above the recovery target, isolate/invalidate that replica and let re-replication replace it
4. Retry lease recovery once racers settle: 'hdfs debug recoverLease'

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
long gs = (r != null) ? r.getGenerationStamp() : -1L;
if (gs < b.getGenerationStamp() || gs > newGS) {
  throw new ReplicaNotFoundException(
      "replica GS " + gs + " outside [" + b.getGenerationStamp() + ", " + newGS + "]");
}
fsDataset.recoverAppend(b, newGS, expectedBlockLen);

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage().contains(ReplicaNotFoundException.UNEXPECTED_GS_REPLICA)) {
    refreshLocatedBlockAndRetryRecoveryOnce();
  } else { throw rnfe; }
}

When it happens

Trigger: recoverAppend/recoverClose during lease recovery where the DataNode replica's GS is below the block's GS (stale replica from an earlier incarnation) or above newGS (the replica already advanced past this recovery target, e.g. from a different concurrent recovery).

Common situations: Client using stale located blocks after lease recovery; two recoveries racing with different newGS values; NN/DN generation stamp divergence after misordered recovery commits.

Related errors

• rur.getRecoveryID() != recoveryId = {recoveryId}, rur={rur}
• Replica was found but missing fields.
• Replica gen stamp < block genstamp, block={block}, replica={
• ProvidedReplica does not yet support writes
• The new recovery id: {} must be greater than the current one

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b389e56b77e8fb38. Report an issue: GitHub.