{"record":{"id":"156db717059913b7","repo":"apache/hadoop","slug":"the-new-generation-stamp-newgs-should-be-greater","errorCode":null,"errorMessage":"The new generation stamp {newGS} should be greater than the replica {b}'s generation stamp","messagePattern":"The new generation stamp (.+?) should be greater than the replica (.+?)'s generation stamp","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1414,"sourceCode":"    }\n  }\n\n\n  @Override  // FsDatasetSpi\n  public ReplicaHandler append(ExtendedBlock b,\n      long newGS, long expectedBlockLen) throws IOException {\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n        b.getBlockPoolId(), getStorageUuidForLock(b),\n        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n      // If the block was successfully finalized because all packets\n      // were successfully processed at the Datanode but the ack for\n      // some of the packets were not received by the client. The client\n      // re-opens the connection and retries sending those packets.\n      // The other reason is that an \"append\" is occurring to this block.\n\n      // check the validity of the parameter\n      if (newGS < b.getGenerationStamp()) {\n        throw new IOException(\"The new generation stamp \" + newGS +\n            \" should be greater than the replica \" + b + \"'s generation stamp\");\n      }\n      ReplicaInfo replicaInfo = getReplicaInfo(b);\n      LOG.info(\"Appending to \" + replicaInfo);\n      if (replicaInfo.getState() != ReplicaState.FINALIZED) {\n        throw new ReplicaNotFoundException(\n            ReplicaNotFoundException.UNFINALIZED_REPLICA + b);\n      }\n      if (replicaInfo.getNumBytes() != expectedBlockLen) {\n        throw new IOException(\"Corrupted replica \" + replicaInfo +\n            \" with a length of \" + replicaInfo.getNumBytes() +\n            \" expected length is \" + expectedBlockLen);\n      }\n\n      FsVolumeReference ref = replicaInfo.getVolume().obtainReference();\n      ReplicaInPipeline replica = null;\n      try {\n        replica = append(b.getBlockPoolId(), replicaInfo, newGS,","sourceCodeStart":1396,"sourceCodeEnd":1432,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1396-L1432","documentation":"Thrown by FsDatasetImpl.append(ExtendedBlock, newGS, expectedBlockLen) when a DataNode is asked to open an append pipeline with a new generation stamp lower than the stamp carried by the block itself (newGS < b.getGenerationStamp()). A generation stamp names one incarnation of a block, and every append must move to a newer stamp so datanodes holding older incarnations can reject stale writers. The DataNode refuses the append because the request targets an older incarnation than the one the caller itself advertised.","triggerScenarios":"BlockReceiver.java:235 calls datanode.data.append(block, newGs, minBytesRcvd) when a client sets up an append pipeline (OP_WRITE_BLOCK after the client bumped the GS via updateBlockForPipeline). The error fires when the newGs derived from a stale LocatedBlock is smaller than the GS inside that same ExtendedBlock, i.e. the client replays an append prepared against an old block incarnation after lease recovery or GS extension already happened elsewhere.","commonSituations":"Client appends with a cached LocatedBlock after the lease was recovered by another writer or via recoverLease; DFSOutputStream retrying an old pipeline setup after a long GC pause or HA failover; rare NameNode/DataNode metadata divergence where the block GS advanced beyond what the client sees.","solutions":["Client side: drop cached block locations, call getBlockLocations again and re-open the append with the fresh GS; make sure retries never reuse the ExtendedBlock+GS pair from a failed attempt","Check no other client holds or recovered the lease: run 'hdfs debug recoverLease -path <file> -reclaim 1', then retry the append","Run 'hdfs fsck <file> -files -blocks -locations' and compare the NameNode block GS with each DataNode replica","If one DataNode replica carries a spuriously higher GS, inspect that DataNode's storage; as a last resort invalidate the bad replica so it is re-replicated from healthy copies"],"exampleFix":"// before: append using a stale LocatedBlock captured before lease recovery\nLocatedBlock stale = client.getLocatedBlocks(file, 0, len).get(0);\ndn.append(stale.getBlock(), stale.getBlock().getGenerationStamp() - 1, len);\n// -> IOException: The new generation stamp ... should be greater ...\n\n// after: refresh the LocatedBlock, then pass a newGS >= block GS (client gets it\n// from updateBlockForPipeline before opening the pipeline)\nLocatedBlock fresh = client.getLocatedBlocks(file, 0, len).get(0);\nlong newGS = client.getNamenode().updateBlockForPipeline(\n    fresh.getBlock(), client.getClientName()).getBlock().getGenerationStamp();\ndn.append(fresh.getBlock(), newGS, len);","handlingStrategy":"validation","validationCode":"if (newGS < b.getGenerationStamp()) {\n  throw new IllegalArgumentException(\"newGS \" + newGS\n      + \" must be >= block GS \" + b.getGenerationStamp() + \"; refresh the LocatedBlock\");\n}\nfsDataset.append(b, newGS, expectedBlockLen);","typeGuard":null,"tryCatchPattern":"catch (IOException ioe) {\n  if (ioe.getMessage() != null && ioe.getMessage().contains(\"generation stamp\")) {\n    refreshLocatedBlocksAndRetryAppendOnce();\n  } else { throw ioe; }\n}","preventionTips":["Re-read block locations (getLocatedBlocks) immediately before opening an append pipeline instead of caching LocatedBlock across RPC failures","Bump the GS via updateBlockForPipeline for every append attempt; never reuse the ExtendedBlock+GS pair from a previous failed attempt","Recover or release the file lease before handing the file to another writer"],"tags":["hdfs","datanode","append","generation-stamp","lease-recovery"],"backgroundTag":"hdfs-generation-stamp-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}