apache/hadoop · error · IOException

THIS IS NOT SUPPOSED TO HAPPEN: replica.getGenerationStamp()

Error message

THIS IS NOT SUPPOSED TO HAPPEN: replica.getGenerationStamp() >= recoveryId = {recoveryId}, block={block}, replica={replica}

What it means

IOException ('THIS IS NOT SUPPOSED TO HAPPEN') thrown by FsDatasetImpl.initReplicaRecoveryImpl when the local replica's generation stamp is already >= the recoveryId of the incoming recovery request. By contract the recoveryId (new GS for the block after recovery) must be strictly greater than every replica's GS, so this signals the NameNode computed an invalid recovery id or the request is stale/replayed.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:3107

      if (replica.getBytesOnDisk() < replica.getVisibleLength()) {
        throw new IOException("getBytesOnDisk() < getVisibleLength(), rip="
            + replica);
      }

      //check the replica's files
      checkReplicaFiles(replica);
    }

    //check generation stamp
    if (replica.getGenerationStamp() < block.getGenerationStamp()) {
      throw new IOException(
          "replica.getGenerationStamp() < block.getGenerationStamp(), block="
          + block + ", replica=" + replica);
    }

    //check recovery id
    if (replica.getGenerationStamp() >= recoveryId) {
      throw new IOException("THIS IS NOT SUPPOSED TO HAPPEN:"
          + " replica.getGenerationStamp() >= recoveryId = " + recoveryId
          + ", block=" + block + ", replica=" + replica);
    }

    //check RUR
    final ReplicaInfo rur;
    if (replica.getState() == ReplicaState.RUR) {
      rur = replica;
      if (rur.getRecoveryID() >= recoveryId) {
        throw new RecoveryInProgressException(
            "rur.getRecoveryID() >= recoveryId = " + recoveryId
            + ", block=" + block + ", rur=" + rur);
      }
      final long oldRecoveryID = rur.getRecoveryID();
      rur.setRecoveryID(recoveryId);
      LOG.info("initReplicaRecovery: update recovery id for " + block
          + " from " + oldRecoveryID + " to " + recoveryId);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the file operation: a fresh lease-recovery attempt makes the NameNode compute a new, higher recoveryId.
2. If seen after an HA failover, verify both NameNodes are healthy and the active is the one issuing recovery (no split-brain).
3. Check for rogue processes speaking the data transfer protocol directly to DataNodes.
4. Persistent occurrence on stock client code is a NameNode bug - collect NN/DN logs and open a HDFS JIRA.

boolean isInvalidRecoveryId(IOException e) {
  return e.getMessage() != null && e.getMessage().contains("THIS IS NOT SUPPOSED TO HAPPEN")
      && e.getMessage().contains("recoveryId");
}

try {
  dataset.initReplicaRecovery(rBlock);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains(">= recoveryId")) {
    // stale/invalid recovery request: re-request recovery so NN mints a higher recoveryId
    namenodeClient.recoverLease(filePath);
    return retryRecoveryLater(rBlock);
  }
  throw e;
}

When it happens

Trigger: initReplicaRecovery(block, recoveryId) where recoveryId <= replica.getGenerationStamp() - e.g., the NameNode dispatched recovery with an id from before a GS bump, or two recovery coordinators (lease recovery + sync) issued overlapping requests with the wrong ordering.

Common situations: Zombie/stale recovery task replayed after a newer recovery already bumped the GS; NameNode state inconsistency after failover (standby promoted with stale pending recovery); custom client code issuing block recovery protocol requests directly.

Related errors

• Replica gen stamp < block genstamp, block={block}, replica={
• ProvidedReplica does not yet support writes
• The new recovery id: {} must be greater than the current one
• Cannot append to a replica with unexpected generation stamp
• Cannot append to a replica with unexpected generation stamp

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/908c1952b7a542c5. Report an issue: GitHub.