apache/hadoop · error · IOException

Generation Stamp should be monotonically increased bpid: {bp

Error message

Generation Stamp should be monotonically increased bpid: {bpid}, block: {replicaInfo}

What it means

IOException thrown by FsDatasetImpl.finalizeReplica when the replica currently registered in the volumeMap has a generation stamp strictly greater than the replica about to be finalized. Generation stamps must be monotonically increasing, so finalizing an older GS over a newer one would silently revert data. This check compares volumeMap.get(bpid, blockId) with the passed-in replicaInfo.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:2049

    if (fsyncDir && finalizedReplicaInfo instanceof FinalizedReplica
        && replicaInfo instanceof LocalReplica) {
      FinalizedReplica finalizedReplica =
          (FinalizedReplica) finalizedReplicaInfo;
      finalizedReplica.fsyncDirectory();
      LocalReplica localReplica = (LocalReplica) replicaInfo;
      localReplica.fsyncDirectory();
    }
  }

  private ReplicaInfo finalizeReplica(String bpid, ReplicaInfo replicaInfo)
      throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        bpid, replicaInfo.getStorageUuid(),
        datasetSubLockStrategy.blockIdToSubLock(replicaInfo.getBlockId()))) {
      // Compare generation stamp of old and new replica before finalizing
      if (volumeMap.get(bpid, replicaInfo.getBlockId()).getGenerationStamp()
          > replicaInfo.getGenerationStamp()) {
        throw new IOException("Generation Stamp should be monotonically "
            + "increased bpid: " + bpid + ", block: " + replicaInfo);
      }

      ReplicaInfo newReplicaInfo = null;
      if (replicaInfo.getState() == ReplicaState.RUR &&
          replicaInfo.getOriginalReplica().getState()
          == ReplicaState.FINALIZED) {
        newReplicaInfo = replicaInfo.getOriginalReplica();
        ((FinalizedReplica)newReplicaInfo).loadLastPartialChunkChecksum();
      } else {
        FsVolumeImpl v = (FsVolumeImpl)replicaInfo.getVolume();
        if (v == null) {
          throw new IOException("No volume for bpid: " + bpid + ", block: " + replicaInfo);
        }

        newReplicaInfo = v.addFinalizedBlock(
            bpid, replicaInfo, replicaInfo, replicaInfo.getBytesReserved());
        if (replicaInfo instanceof ReplicaInPipeline) {

View on GitHub (pinned to 2add963021)

Solutions

1. Let the lease recovery complete and have the client re-open the file - the newer GS replica wins by design.
2. Verify there is only one active writer per file (no dual clients bypassing the lease, e.g., two processes with dfs.client.use.datanode.hostname style raw access or append after lease steal).
3. Check NameNode lease tables (hdfs dfsadmin -metasave) for conflicting lease holders on the same file.
4. If it persists after all writers stopped, restart the DataNode to rebuild the volume map.

try {
  dataset.finalizeBlock(b, fsyncDir);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Generation Stamp should be monotonically")) {
    // a newer-GS replica already won; abandon this finalize and let the client re-open
    LOG.warn("Lost finalize race on " + b + "; newer replica exists", e);
    return;
  }
  throw e;
}

When it happens

Trigger: finalizeReplica invoked (from finalizeBlock or RUR recovery finalization) when, between the caller reading the replica and finalizing it, another thread (lease recovery / updateReplica bumping GS) installed a newer-generation replica in the volumeMap for the same block id.

Common situations: Concurrent lease recovery and client close racing on the same block; two block-recovery sessions with different recovery ids; DataNode processing an OP_WRITE_BLOCK recovery while a finalize is queued.

Related errors

• Cannot append to a replica with unexpected generation stamp
• The new generation stamp {newGS} should be greater than the
• Cannot append to a replica with unexpected generation stamp
• rur.getRecoveryID() != recoveryId = {recoveryId}, rur={rur}
• Replica was found but missing fields.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e70a4bf4cb6227fc. Report an issue: GitHub.