{"record":{"id":"f44859848d724b34","repo":"apache/hadoop","slug":"commit-block-with-mismatching-gs-nn-has-block","errorCode":null,"errorMessage":"Commit block with mismatching GS. NN has {block}, client submits {commitBlock}","messagePattern":"Commit block with mismatching GS\\. NN has (.+?), client submits (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java","lineNumber":1192,"sourceCode":"\n /**\n * Commit a block of a file\n * \n * @param block block to be committed\n * @param commitBlock - contains client reported block length and generation\n * @return true if the block is changed to committed state.\n * @throws IOException if the block does not have at least a minimal number\n * of replicas reported from data-nodes.\n */\n private boolean commitBlock(final BlockInfo block,\n final Block commitBlock) throws IOException {\n if (block.getBlockUCState() == BlockUCState.COMMITTED)\n return false;\n assert block.getNumBytes() <= commitBlock.getNumBytes() :\n \"commitBlock length is less than the stored one \"\n + commitBlock.getNumBytes() + \" vs. \" + block.getNumBytes();\n if(block.getGenerationStamp() != commitBlock.getGenerationStamp()) {\n throw new IOException(\"Commit block with mismatching GS. NN has \" +\n block + \", client submits \" + commitBlock);\n }\n List staleReplicas =\n block.commitBlock(commitBlock);\n removeStaleReplicas(staleReplicas, block);\n return true;\n }\n \n /**\n * Commit the last block of the file and mark it as complete if it has\n * meets the minimum redundancy requirement\n * \n * @param bc block collection\n * @param commitBlock - contains client reported block length and generation\n * @param iip - INodes in path to bc\n * @return true if the last block is changed to committed state.\n * @throws IOException if the block does not have at least a minimal number\n * of replicas reported from data-nodes.","sourceCodeStart":1174,"sourceCodeEnd":1210,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L1174-L1210","documentation":"Thrown by NameNode BlockManager.commitBlock when a client finalizes the last block of a file but the generation stamp (GS) in the client's commit request does not match the GS the NameNode has recorded for that block. The NameNode tracks every block write/lease-recovery by bumping the block's generation stamp; a mismatch means the client is committing a different block instance than the one the NameNode believes is under construction. This usually indicates that lease recovery or another writer bumped the GS after this client started writing.","triggerScenarios":"Client calls FSNamespaceOp/ClientProtocol.completeFile (or addBlock during retry) with a commitBlock whose getGenerationStamp() differs from block.getGenerationStamp() held by the NN; typically after lease recovery, block recovery by another client/DN, or a client retrying a write against stale block state. Also occurs when a client recovers a file (truncate/append path) and the recovered GS is not used in the subsequent commit.","commonSituations":"Two clients writing the same file (soft lease expired and lease taken over); a long-running writer paused (GC pause, network partition) while its lease was recovered; HDFS client version mismatch in recovery protocols; appends after a crash where recovered GS was not picked up.","solutions":["Close/reopen the file: let the failed writer abort and have one client re-acquire the lease and continue with the block state the NN reports (recoverFileLease then append)","Call ClientProtocol.recoverLease on the file before writing/appending so your client works with the post-recovery generation stamp","Ensure only one client holds the write lease at a time (check LeaseManager via fsck/`hdfs dfsadmin` output or NN logs for concurrent lease recovery)","Upgrade mismatched HDFS client/hadoop-distcp versions involved in the write so recovery handshake is consistent"],"exampleFix":"// before: retry completeFile with stale block GS after lease was recovered\nfs.complete(file, newBlock); // newBlock GS != NN GS -> IOException mismatching GS\n\n// after: recover lease first, then append/complete with NN's block state\nif (!fs.append(file).hasNullBlock()) { /* re-acquire */ }\nClientProtocol cp = ((DistributedFileSystem) fs).getClient().getNamenode();\ncp.recoverLease(file.toString(), clientName);\n// re-fetch file status, re-open for append, write, then complete","handlingStrategy":"retry","validationCode":"// Before completing: recover the lease so client and NN agree on the last block's GS\nClientProtocol nn = ((DistributedFileSystem) fs).getClient().getNamenode();\nnn.recoverLease(src, clientName); // returns true when file is closed or client owns lease\nHdfsDataInputStream in = (HdfsDataInputStream) fs.open(src);\nLocatedBlock lb = in.getCurrentBlockLen() >= 0 ? in.readLocatedBlockInfo()[0] : null;\n// use lb's block id + GS from NN when issuing complete","typeGuard":null,"tryCatchPattern":"try {\n boolean closed = nn.complete(src, clientName, lastBlockFromNN, iip);\n} catch (IOException e) {\n if (e.getMessage().contains(\"mismatching GS\")) {\n nn.recoverLease(src, clientName); // adopt NN block state, then re-append and complete\n } else { throw e; }\n}","preventionTips":["Hold the write lease for the whole write; avoid concurrent writers on one file","Call hflush/hsync periodically so client and NN block state stay aligned","On any write failure, run recoverLease before re-opening for append","Keep HDFS client version aligned with the cluster version"],"tags":["hdfs","namenode","block","generation-stamp","lease-recovery","write-conflict"],"backgroundTag":"generation-stamp-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}