apache/hadoop · error · IOException

Generation stamps mismatched: block${i}=${blocki}, expected

Error message

Generation stamps mismatched: block${i}=${blocki}, expected block group=${blockGroup}

What it means

Third invariant in StripedBlockUtil.checkBlocks(): every internal block i of a striped group must carry the same generation stamp as the group. blocki.getGenerationStamp() != blockGroup.getGenerationStamp() throws IOException from DFSStripedOutputStream.waitEndBlocks, aborting the erasure-coded write. Generation stamps are bumped during block recovery, so a mismatch means one internal block was recovered independently of the group view.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/StripedBlockUtil.java:1018

    }
  }

  /**
   * Check if the information such as IDs and generation stamps in block-i
   * match the block group.
   */
  public static void checkBlocks(ExtendedBlock blockGroup,
      int i, ExtendedBlock blocki) throws IOException {
    if (!blocki.getBlockPoolId().equals(blockGroup.getBlockPoolId())) {
      throw new IOException("Block pool IDs mismatched: block" + i + "="
          + blocki + ", expected block group=" + blockGroup);
    }
    if (blocki.getBlockId() - i != blockGroup.getBlockId()) {
      throw new IOException("Block IDs mismatched: block" + i + "="
          + blocki + ", expected block group=" + blockGroup);
    }
    if (blocki.getGenerationStamp() != blockGroup.getGenerationStamp()) {
      throw new IOException("Generation stamps mismatched: block" + i + "="
          + blocki + ", expected block group=" + blockGroup);
    }
  }

  public static int getBlockIndex(Block reportedBlock) {
    long BLOCK_GROUP_INDEX_MASK = 15;
    return (int) (reportedBlock.getBlockId() &
        BLOCK_GROUP_INDEX_MASK);
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Retry the write — after recovery completes, a fresh block group gets consistent stamps
Check DataNode health and network stability; repeated mid-write DN failures keep re-triggering recovery
Verify uniform Hadoop versions (generation stamp propagation for striped groups changed across releases)
If a single DN's failure reliably reproduces it, capture its logs plus the stamps from the message and report upstream

Defensive patterns

Strategy: try-catch

Try / catch

try {
  try (FSDataOutputStream out = fs.create(ecPath)) { /* write striped data */ }
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("Generation stamps mismatched")) {
    // striped recovery left inconsistent stamps; retry gets a fresh block group
    LOG.warn("Striped write failed on generation stamp mismatch, retrying", e);
  } else { throw e; }
}

Prevention

Keep DataNodes stable during large EC writes to avoid mid-write recovery
Monitor DN restart/decommission events when striped writes fail repeatedly

When it happens

Trigger: Writing an EC file when a datanode failure triggers pipeline recovery that rolls an internal block's generation stamp while the writer still compares against the original group stamp — recovery did not propagate the new stamp to the whole group view.

Common situations: Struggles during rolling upgrades or datanode restarts mid-write of striped files; historically associated with striped-pipeline recovery bugs. Users see the EC write fail after a DN drop.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ba5215ad02c7cb89. Report an issue: GitHub.