apache/hadoop · critical · IOException

Trying to commit inconsistent block: id = {blockId}, expecte

Error message

Trying to commit inconsistent block: id = {blockId}, expected id = {expectedId}

What it means

BlockInfo.commitBlock is invoked by the NameNode when a client commits the last block of a file (close/append completion): the client-reported Block must carry the same block ID as the stored under-construction BlockInfo. A mismatch throws IOException('Trying to commit inconsistent block: id = X, expected id = Y') before any metadata is mutated. This is a fail-fast integrity guard against NameNode namespace/block state divergence - essentially never expected on a healthy cluster.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java:444

  public List<ReplicaUnderConstruction> setGenerationStampAndVerifyReplicas(
      long genStamp) {
    Preconditions.checkState(uc != null && !isComplete());
    // Set the generation stamp for the block.
    setGenerationStamp(genStamp);

    return uc.getStaleReplicas(genStamp);
  }

  /**
   * Commit block's length and generation stamp as reported by the client.
   * Set block state to {@link BlockUCState#COMMITTED}.
   * @param block - contains client reported block length and generation
   * @return staleReplica's List.
   * @throws IOException if block ids are inconsistent.
   */
  List<ReplicaUnderConstruction> commitBlock(Block block) throws IOException {
    if (getBlockId() != block.getBlockId()) {
      throw new IOException("Trying to commit inconsistent block: id = "
          + block.getBlockId() + ", expected id = " + getBlockId());
    }
    Preconditions.checkState(!isComplete());
    uc.commit();
    this.setNumBytes(block.getNumBytes());
    // Sort out invalid replicas.
    return setGenerationStampAndVerifyReplicas(block.getGenerationStamp());
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Capture the NameNode log around the throw (file path, block id, expected id) and retry the client operation - many one-off occurrences are benign recovery races
Run hdfs fsck on the affected file/path to check block consistency
If persistent, search/report an HDFS JIRA with the log excerpt and stack; the file may need to be deleted or recovered from backup, since this signals namespace-level inconsistency

Defensive patterns

Strategy: try-catch

Try / catch

try { fs.close(); / fs.append(...) }
catch (IOException e) {
  if (e.getMessage().startsWith("Trying to commit inconsistent block")) { captureNNLog(); runFsck(path); reportToCommunity(e); }
  throw e;
}

Prevention

Keep NameNodes on patched releases - occurrences of this error usually map to fixed lease-recovery bugs
Never force-start a NameNode on inconsistent edits/images; keep reliable checkpoints/fsimage backups
On first occurrence, capture the block/file IDs and hdfs fsck output before retrying writes to that file

When it happens

Trigger: Lease recovery, append or file-close paths where the stored BlockUC block ID differs from what the client reports - caused by NameNode bugs, corrupted/inconsistent edit logs or fsimage, downgrade/upgrade edge cases, or races between multiple writers and recovery.

Common situations: Rare; seen around HDFS bug reports involving lease recovery races or block-recovery replay. If it recurs for the same file, the file's block metadata is suspect. Not caused by user configuration.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/21d6658aadf823e2. Report an issue: GitHub.