{"record":{"id":"e70a4bf4cb6227fc","repo":"apache/hadoop","slug":"generation-stamp-should-be-monotonically-increased","errorCode":null,"errorMessage":"Generation Stamp should be monotonically increased bpid: {bpid}, block: {replicaInfo}","messagePattern":"Generation Stamp should be monotonically increased bpid: (.+?), block: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":2049,"sourceCode":"    if (fsyncDir && finalizedReplicaInfo instanceof FinalizedReplica\n        && replicaInfo instanceof LocalReplica) {\n      FinalizedReplica finalizedReplica =\n          (FinalizedReplica) finalizedReplicaInfo;\n      finalizedReplica.fsyncDirectory();\n      LocalReplica localReplica = (LocalReplica) replicaInfo;\n      localReplica.fsyncDirectory();\n    }\n  }\n\n  private ReplicaInfo finalizeReplica(String bpid, ReplicaInfo replicaInfo)\n      throws IOException {\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n        bpid, replicaInfo.getStorageUuid(),\n        datasetSubLockStrategy.blockIdToSubLock(replicaInfo.getBlockId()))) {\n      // Compare generation stamp of old and new replica before finalizing\n      if (volumeMap.get(bpid, replicaInfo.getBlockId()).getGenerationStamp()\n          > replicaInfo.getGenerationStamp()) {\n        throw new IOException(\"Generation Stamp should be monotonically \"\n            + \"increased bpid: \" + bpid + \", block: \" + replicaInfo);\n      }\n\n      ReplicaInfo newReplicaInfo = null;\n      if (replicaInfo.getState() == ReplicaState.RUR &&\n          replicaInfo.getOriginalReplica().getState()\n          == ReplicaState.FINALIZED) {\n        newReplicaInfo = replicaInfo.getOriginalReplica();\n        ((FinalizedReplica)newReplicaInfo).loadLastPartialChunkChecksum();\n      } else {\n        FsVolumeImpl v = (FsVolumeImpl)replicaInfo.getVolume();\n        if (v == null) {\n          throw new IOException(\"No volume for bpid: \" + bpid + \", block: \" + replicaInfo);\n        }\n\n        newReplicaInfo = v.addFinalizedBlock(\n            bpid, replicaInfo, replicaInfo, replicaInfo.getBytesReserved());\n        if (replicaInfo instanceof ReplicaInPipeline) {","sourceCodeStart":2031,"sourceCodeEnd":2067,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L2031-L2067","documentation":"IOException thrown by FsDatasetImpl.finalizeReplica when the replica currently registered in the volumeMap has a generation stamp strictly greater than the replica about to be finalized. Generation stamps must be monotonically increasing, so finalizing an older GS over a newer one would silently revert data. This check compares volumeMap.get(bpid, blockId) with the passed-in replicaInfo.","triggerScenarios":"finalizeReplica invoked (from finalizeBlock or RUR recovery finalization) when, between the caller reading the replica and finalizing it, another thread (lease recovery / updateReplica bumping GS) installed a newer-generation replica in the volumeMap for the same block id.","commonSituations":"Concurrent lease recovery and client close racing on the same block; two block-recovery sessions with different recovery ids; DataNode processing an OP_WRITE_BLOCK recovery while a finalize is queued.","solutions":["Let the lease recovery complete and have the client re-open the file - the newer GS replica wins by design.","Verify there is only one active writer per file (no dual clients bypassing the lease, e.g., two processes with dfs.client.use.datanode.hostname style raw access or append after lease steal).","Check NameNode lease tables (hdfs dfsadmin -metasave) for conflicting lease holders on the same file.","If it persists after all writers stopped, restart the DataNode to rebuild the volume map."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  dataset.finalizeBlock(b, fsyncDir);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Generation Stamp should be monotonically\")) {\n    // a newer-GS replica already won; abandon this finalize and let the client re-open\n    LOG.warn(\"Lost finalize race on \" + b + \"; newer replica exists\", e);\n    return;\n  }\n  throw e;\n}","preventionTips":["Serialize writers per file through HDFS leases; never run two appenders on one file.","After lease recovery, re-open the file instead of continuing with pre-recovery block handles.","Monitor for concurrent-recovery warnings in NameNode logs to catch dual-coordinator setups."],"tags":["hdfs","datanode","finalize","generation-stamp","lease-recovery"],"backgroundTag":"generation-stamp-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}