apache/hadoop · critical · IOException

Mismatched block IDs or generation stamps, attempting to rep

Error message

Mismatched block IDs or generation stamps, attempting to replace block {} with {} as block # {}/{} of {}

What it means

updateBlocks aligns a file's existing block list with the new list carried by an OP_ADD, OP_CLOSE, or OP_SET_BLOCKS op. For shared indices the block ids must match, and generation stamps must match except on the last block when the op is a pure generation-stamp update (equal-length lists). Any other mismatch throws IOException with both blocks, the index, and the path.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java:1147

      throws IOException {
    // Update its block list
    BlockInfo[] oldBlocks = file.getBlocks();
    Block[] newBlocks = op.getBlocks();
    String path = op.getPath();
    
    // Are we only updating the last block's gen stamp.
    boolean isGenStampUpdate = oldBlocks.length == newBlocks.length;
    
    // First, update blocks in common
    for (int i = 0; i < oldBlocks.length && i < newBlocks.length; i++) {
      BlockInfo oldBlock = oldBlocks[i];
      Block newBlock = newBlocks[i];
      
      boolean isLastBlock = i == newBlocks.length - 1;
      if (oldBlock.getBlockId() != newBlock.getBlockId() ||
          (oldBlock.getGenerationStamp() != newBlock.getGenerationStamp() && 
              !(isGenStampUpdate && isLastBlock))) {
        throw new IOException("Mismatched block IDs or generation stamps, " +
            "attempting to replace block " + oldBlock + " with " + newBlock +
            " as block # " + i + "/" + newBlocks.length + " of " +
            path);
      }
      
      oldBlock.setNumBytes(newBlock.getNumBytes());
      boolean changeMade =
        oldBlock.getGenerationStamp() != newBlock.getGenerationStamp();
      final long newGenerationStamp = newBlock.getGenerationStamp();
      oldBlock.setGenerationStamp(newGenerationStamp);
      // Update global generation stamp in Standby NameNode
      fsNamesys.getBlockManager().getBlockIdManager().
          setGenerationStampIfGreater(newGenerationStamp);

      if (!oldBlock.isComplete() &&
          (!isLastBlock || op.shouldCompleteLastBlock())) {
        changeMade = true;
        fsNamesys.getBlockManager().forceCompleteBlock(oldBlock);

View on GitHub (pinned to 2add963021)

Solutions

1. Restore a consistent fsimage plus edits from one checkpoint epoch
2. Run 'hdfs namenode -recover' to skip the inconsistent section
3. Audit all journal copies (JournalNodes, NFS) for duplicated, overlapping, or truncated segments

Example fix

# before: replay halts on 'Mismatched block IDs ... attempting to replace block'
hdfs --daemon start namenode

# after: verify the segment chain, then recover
hdfs oev -i <suspect-segment> -o /tmp/check.xml -p xml   # confirm parse failure
hdfs namenode -recover

try {
  loader.loadFSEdits(storage, 0);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("attempting to replace block")) {
    // block list prefix disagrees with namespace state: single-epoch restore
    // or 'hdfs namenode -recover'
  }
  throw e;
}

When it happens

Trigger: Replaying a block-list update whose prefix disagrees with the file's current blocks: image and edits from different generations, the same update replayed twice, or corrupted block records in the op.

Common situations: Same family as other replay inconsistencies: mixed backup files, duplicated segments, post-crash corruption.

Related errors

• Mismatched block IDs or generation stamps for the old last b
• Trying to remove more than one block from file {}
• Trying to delete non-existant block {}
• The layout version {} supports inodeId but gave bogus inodeI
• Trying to remove a block from file {} which is not under con

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/a052a838939a88d9. Report an issue: GitHub.