apache/hadoop · error · IOException

The new generation stamp {newGS} should be greater than the

Error message

The new generation stamp {newGS} should be greater than the replica {b}'s generation stamp

What it means

Thrown by FsDatasetImpl.append(ExtendedBlock, newGS, expectedBlockLen) when a DataNode is asked to open an append pipeline with a new generation stamp lower than the stamp carried by the block itself (newGS < b.getGenerationStamp()). A generation stamp names one incarnation of a block, and every append must move to a newer stamp so datanodes holding older incarnations can reject stale writers. The DataNode refuses the append because the request targets an older incarnation than the one the caller itself advertised.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1414

    }
  }


  @Override  // FsDatasetSpi
  public ReplicaHandler append(ExtendedBlock b,
      long newGS, long expectedBlockLen) throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        b.getBlockPoolId(), getStorageUuidForLock(b),
        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
      // If the block was successfully finalized because all packets
      // were successfully processed at the Datanode but the ack for
      // some of the packets were not received by the client. The client
      // re-opens the connection and retries sending those packets.
      // The other reason is that an "append" is occurring to this block.

      // check the validity of the parameter
      if (newGS < b.getGenerationStamp()) {
        throw new IOException("The new generation stamp " + newGS +
            " should be greater than the replica " + b + "'s generation stamp");
      }
      ReplicaInfo replicaInfo = getReplicaInfo(b);
      LOG.info("Appending to " + replicaInfo);
      if (replicaInfo.getState() != ReplicaState.FINALIZED) {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.UNFINALIZED_REPLICA + b);
      }
      if (replicaInfo.getNumBytes() != expectedBlockLen) {
        throw new IOException("Corrupted replica " + replicaInfo +
            " with a length of " + replicaInfo.getNumBytes() +
            " expected length is " + expectedBlockLen);
      }

      FsVolumeReference ref = replicaInfo.getVolume().obtainReference();
      ReplicaInPipeline replica = null;
      try {
        replica = append(b.getBlockPoolId(), replicaInfo, newGS,

View on GitHub (pinned to 2add963021)

Solutions

1. Client side: drop cached block locations, call getBlockLocations again and re-open the append with the fresh GS; make sure retries never reuse the ExtendedBlock+GS pair from a failed attempt
2. Check no other client holds or recovered the lease: run 'hdfs debug recoverLease -path <file> -reclaim 1', then retry the append
3. Run 'hdfs fsck <file> -files -blocks -locations' and compare the NameNode block GS with each DataNode replica
4. If one DataNode replica carries a spuriously higher GS, inspect that DataNode's storage; as a last resort invalidate the bad replica so it is re-replicated from healthy copies

Example fix

// before: append using a stale LocatedBlock captured before lease recovery
LocatedBlock stale = client.getLocatedBlocks(file, 0, len).get(0);
dn.append(stale.getBlock(), stale.getBlock().getGenerationStamp() - 1, len);
// -> IOException: The new generation stamp ... should be greater ...

// after: refresh the LocatedBlock, then pass a newGS >= block GS (client gets it
// from updateBlockForPipeline before opening the pipeline)
LocatedBlock fresh = client.getLocatedBlocks(file, 0, len).get(0);
long newGS = client.getNamenode().updateBlockForPipeline(
    fresh.getBlock(), client.getClientName()).getBlock().getGenerationStamp();
dn.append(fresh.getBlock(), newGS, len);

if (newGS < b.getGenerationStamp()) {
  throw new IllegalArgumentException("newGS " + newGS
      + " must be >= block GS " + b.getGenerationStamp() + "; refresh the LocatedBlock");
}
fsDataset.append(b, newGS, expectedBlockLen);

catch (IOException ioe) {
  if (ioe.getMessage() != null && ioe.getMessage().contains("generation stamp")) {
    refreshLocatedBlocksAndRetryAppendOnce();
  } else { throw ioe; }
}

When it happens

Trigger: BlockReceiver.java:235 calls datanode.data.append(block, newGs, minBytesRcvd) when a client sets up an append pipeline (OP_WRITE_BLOCK after the client bumped the GS via updateBlockForPipeline). The error fires when the newGs derived from a stale LocatedBlock is smaller than the GS inside that same ExtendedBlock, i.e. the client replays an append prepared against an old block incarnation after lease recovery or GS extension already happened elsewhere.

Common situations: Client appends with a cached LocatedBlock after the lease was recovered by another writer or via recoverLease; DFSOutputStream retrying an old pipeline setup after a long GC pause or HA failover; rare NameNode/DataNode metadata divergence where the block GS advanced beyond what the client sees.

Related errors

• Cannot append to a replica with unexpected generation stamp
• Cannot append to an unfinalized replica {b}
• Cannot append to a replica with unexpected generation stamp
• Generation Stamp should be monotonically increased bpid: {bp
• rur.getRecoveryID() != recoveryId = {recoveryId}, rur={rur}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/156db717059913b7. Report an issue: GitHub.