apache/hadoop · error · IOException

Commit block with mismatching GS. NN has {block}, client sub

Error message

Commit block with mismatching GS. NN has {block}, client submits {commitBlock}

What it means

Thrown by NameNode BlockManager.commitBlock when a client finalizes the last block of a file but the generation stamp (GS) in the client's commit request does not match the GS the NameNode has recorded for that block. The NameNode tracks every block write/lease-recovery by bumping the block's generation stamp; a mismatch means the client is committing a different block instance than the one the NameNode believes is under construction. This usually indicates that lease recovery or another writer bumped the GS after this client started writing.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:1192

  /**
   * Commit a block of a file
   * 
   * @param block block to be committed
   * @param commitBlock - contains client reported block length and generation
   * @return true if the block is changed to committed state.
   * @throws IOException if the block does not have at least a minimal number
   * of replicas reported from data-nodes.
   */
  private boolean commitBlock(final BlockInfo block,
      final Block commitBlock) throws IOException {
    if (block.getBlockUCState() == BlockUCState.COMMITTED)
      return false;
    assert block.getNumBytes() <= commitBlock.getNumBytes() :
        "commitBlock length is less than the stored one "
            + commitBlock.getNumBytes() + " vs. " + block.getNumBytes();
    if(block.getGenerationStamp() != commitBlock.getGenerationStamp()) {
      throw new IOException("Commit block with mismatching GS. NN has " +
          block + ", client submits " + commitBlock);
    }
    List<ReplicaUnderConstruction> staleReplicas =
        block.commitBlock(commitBlock);
    removeStaleReplicas(staleReplicas, block);
    return true;
  }
  
  /**
   * Commit the last block of the file and mark it as complete if it has
   * meets the minimum redundancy requirement
   * 
   * @param bc block collection
   * @param commitBlock - contains client reported block length and generation
   * @param iip - INodes in path to bc
   * @return true if the last block is changed to committed state.
   * @throws IOException if the block does not have at least a minimal number
   * of replicas reported from data-nodes.

View on GitHub (pinned to 2add963021)

Solutions

  1. Close/reopen the file: let the failed writer abort and have one client re-acquire the lease and continue with the block state the NN reports (recoverFileLease then append)
  2. Call ClientProtocol.recoverLease on the file before writing/appending so your client works with the post-recovery generation stamp
  3. Ensure only one client holds the write lease at a time (check LeaseManager via fsck/`hdfs dfsadmin` output or NN logs for concurrent lease recovery)
  4. Upgrade mismatched HDFS client/hadoop-distcp versions involved in the write so recovery handshake is consistent

Example fix

// before: retry completeFile with stale block GS after lease was recovered
fs.complete(file, newBlock); // newBlock GS != NN GS -> IOException mismatching GS

// after: recover lease first, then append/complete with NN's block state
if (!fs.append(file).hasNullBlock()) { /* re-acquire */ }
ClientProtocol cp = ((DistributedFileSystem) fs).getClient().getNamenode();
cp.recoverLease(file.toString(), clientName);
// re-fetch file status, re-open for append, write, then complete
Defensive patterns

Strategy: retry

Validate before calling

// Before completing: recover the lease so client and NN agree on the last block's GS
ClientProtocol nn = ((DistributedFileSystem) fs).getClient().getNamenode();
nn.recoverLease(src, clientName); // returns true when file is closed or client owns lease
HdfsDataInputStream in = (HdfsDataInputStream) fs.open(src);
LocatedBlock lb = in.getCurrentBlockLen() >= 0 ? in.readLocatedBlockInfo()[0] : null;
// use lb's block id + GS from NN when issuing complete

Try / catch

try {
  boolean closed = nn.complete(src, clientName, lastBlockFromNN, iip);
} catch (IOException e) {
  if (e.getMessage().contains("mismatching GS")) {
    nn.recoverLease(src, clientName); // adopt NN block state, then re-append and complete
  } else { throw e; }
}

Prevention

When it happens

Trigger: Client calls FSNamespaceOp/ClientProtocol.completeFile (or addBlock during retry) with a commitBlock whose getGenerationStamp() differs from block.getGenerationStamp() held by the NN; typically after lease recovery, block recovery by another client/DN, or a client retrying a write against stale block state. Also occurs when a client recovers a file (truncate/append path) and the recovered GS is not used in the subsequent commit.

Common situations: Two clients writing the same file (soft lease expired and lease taken over); a long-running writer paused (GC pause, network partition) while its lease was recovered; HDFS client version mismatch in recovery protocols; appends after a crash where recovered GS was not picked up.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f44859848d724b34. Report an issue: GitHub.