apache/hadoop · error · ReplicaNotFoundException

Cannot append to a replica with unexpected generation stamp

Error message

Cannot append to a replica with unexpected generation stamp {b}. Expected GS range is [{generationStamp}, {newGS}].

What it means

Same generation-stamp range contract as recoverCheck but for the RBW fast path: recoverRbwImpl requires the RBW replica's GS to lie in [b.getGenerationStamp(), newGS]. Outside that range the DataNode refuses with ReplicaNotFoundException (UNEXPECTED_GS_REPLICA), the message embedding the block b with its stamp.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1749

    } finally {
      if (dataNodeMetrics != null) {
        long recoverRbwMs = Time.monotonicNow() - startTimeMs;
        dataNodeMetrics.addRecoverRbwOp(recoverRbwMs);
      }
    }
  }

  private ReplicaHandler recoverRbwImpl(ReplicaInPipeline rbw,
      ExtendedBlock b, long newGS, long minBytesRcvd, long maxBytesRcvd)
      throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        b.getBlockPoolId(), getStorageUuidForLock(b),
        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
      // check generation stamp
      long replicaGenerationStamp = rbw.getGenerationStamp();
      if (replicaGenerationStamp < b.getGenerationStamp() ||
          replicaGenerationStamp > newGS) {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.UNEXPECTED_GS_REPLICA + b +
                ". Expected GS range is [" + b.getGenerationStamp() + ", " +
                newGS + "].");
      }

      // check replica length
      long bytesAcked = rbw.getBytesAcked();
      long numBytes = rbw.getNumBytes();
      if (bytesAcked < minBytesRcvd || numBytes > maxBytesRcvd) {
        throw new ReplicaNotFoundException("Unmatched length replica " +
            rbw + ": BytesAcked = " + bytesAcked +
            " BytesRcvd = " + numBytes + " are not in the range of [" +
            minBytesRcvd + ", " + maxBytesRcvd + "].");
      }

      long bytesOnDisk = rbw.getBytesOnDisk();
      long blockDataLength = rbw.getReplicaInfo().getBlockDataLength();
      if (bytesOnDisk != blockDataLength) {

View on GitHub (pinned to 2add963021)

Solutions

1. Refresh located blocks and retry with the current GS range
2. Verify the block GS on the NN vs DN logs; a replica persistently above newGS should be invalidated
3. Let one recovery finish before starting the next (lease serialization enforces this)
4. Run 'hdfs fsck' to reconcile block and replica GS state if it loops

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
long gs = (r != null) ? r.getGenerationStamp() : -1L;
if (gs < b.getGenerationStamp() || gs > newGS) {
  refreshLocatedBlocksAndRebuildPipeline(); // GSs are stale somewhere upstream
  return;
}
fsDataset.recoverRbw(b, newGS, minBytesRcvd, maxBytesRcvd);

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage().contains(ReplicaNotFoundException.UNEXPECTED_GS_REPLICA)) {
    refreshLocatedBlocksAndRetryRecoveryOnce();
  } else { throw rnfe; }
}

When it happens

Trigger: Pipeline recovery where this DN's RBW replica carries a GS older than the block's (stale from a previous incarnation) or newer than newGS (a different recovery already advanced it).

Common situations: Client rebuilding a pipeline from stale located blocks after another recovery bumped GSs; concurrent lease and block recovery interleaving; retry storms reusing outdated ExtendedBlocks.

Related errors

• Replica gen stamp < block genstamp, block={block}, replica={
• ProvidedReplica does not yet support writes
• The new recovery id: {} must be greater than the current one
• Cannot append to a replica with unexpected generation stamp
• RBW replica {replicaInfo}bytesRcvd({numBytes}), bytesOnDisk(

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ce5f0cca6def1d80. Report an issue: GitHub.