apache/hadoop · critical · IOException

Mismatched block IDs or generation stamps for the old last b

Error message

Mismatched block IDs or generation stamps for the old last block of file {}, the old last block is {}, and the block read from editlog is {}

What it means

addNewBlock replays OP_ADD_BLOCK: the op's penultimate block must equal the file's current last block in both block id and generation stamp. A mismatch means the namespace's block state diverges from the edit log, and replay stops with IOException naming the file, the old last block, and the block read from the log.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java:1093

    return sb.toString();
  }

  /**
   * Add a new block into the given INodeFile
   */
  private void addNewBlock(AddBlockOp op, INodeFile file,
      ErasureCodingPolicy ecPolicy) throws IOException {
    BlockInfo[] oldBlocks = file.getBlocks();
    Block pBlock = op.getPenultimateBlock();
    Block newBlock= op.getLastBlock();
    
    if (pBlock != null) { // the penultimate block is not null
      assert oldBlocks != null && oldBlocks.length > 0;
      // compare pBlock with the last block of oldBlocks
      BlockInfo oldLastBlock = oldBlocks[oldBlocks.length - 1];
      if (oldLastBlock.getBlockId() != pBlock.getBlockId()
          || oldLastBlock.getGenerationStamp() != pBlock.getGenerationStamp()) {
        throw new IOException(
            "Mismatched block IDs or generation stamps for the old last block of file "
                + op.getPath() + ", the old last block is " + oldLastBlock
                + ", and the block read from editlog is " + pBlock);
      }
      
      oldLastBlock.setNumBytes(pBlock.getNumBytes());
      if (!oldLastBlock.isComplete()) {
        fsNamesys.getBlockManager().forceCompleteBlock(oldLastBlock);
        fsNamesys.getBlockManager().processQueuedMessagesForBlock(pBlock);
      }
    } else { // the penultimate block is null
      Preconditions.checkState(oldBlocks == null || oldBlocks.length == 0);
    }
    // add the new block
    final BlockInfo newBlockInfo;
    boolean isStriped = ecPolicy != null;
    if (isStriped) {
      newBlockInfo = new BlockInfoStriped(newBlock, ecPolicy);

View on GitHub (pinned to 2add963021)

Solutions

1. Restore fsimage and edits from the same checkpoint epoch
2. Recover with 'hdfs namenode -recover', accepting that the skipped transactions are lost
3. Check JournalNode or NFS journal copies for duplicated or truncated segments, then re-bootstrap standbys from the active

Example fix

# before: mixed image and edits replay OP_ADD_BLOCK against stale block state
# IOException: Mismatched block IDs ... for the old last block of file /data/x

# after: single-epoch restore and recovery
rm -rf /hadoop/dfs/name/current && cp -a backup/current /hadoop/dfs/name/current
hdfs namenode -recover   # if the chain is still inconsistent

hdfs oev -i <segment> -o /tmp/check.xml -p xml
# a full parse proves block fields and checksums are intact before replay

try {
  loader.loadFSEdits(storage, 0);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Mismatched block IDs")) {
    // block state diverged from the log: restore a single-epoch image+edits
    // pair, or restart with 'hdfs namenode -recover' and accept the loss
  }
  throw e;
}

When it happens

Trigger: OP_ADD_BLOCK replayed against a file whose block list came from a different image/edits generation; duplicated or out-of-order OP_ADD_BLOCK transactions; corrupted block id or genstamp fields inside the op.

Common situations: fsimage restored from one point in time and edits from another; segments duplicated during manual recovery; disk corruption after a host failure.

Related errors

• Mismatched block IDs or generation stamps, attempting to rep
• Trying to remove more than one block from file {}
• Trying to delete non-existant block {}
• The layout version {} supports inodeId but gave bogus inodeI
• Trying to remove a block from file {} which is not under con

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bee0ec55e65e4f8a. Report an issue: GitHub.