{"record":{"id":"5cddc3b293db4120","repo":"apache/hadoop","slug":"only-a-finalized-replica-can-be-appended-to-repli","errorCode":null,"errorMessage":"Only a Finalized replica can be appended to; Replica with blk id {blockId} has state {state}","messagePattern":"Only a Finalized replica can be appended to; Replica with blk id (.+?) has state (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1462,"sourceCode":"   * bump its generation stamp to be the newGS\n   * \n   * @param bpid block pool Id\n   * @param replicaInfo a finalized replica\n   * @param newGS new generation stamp\n   * @param estimateBlockLen estimate block length\n   * @return a RBW replica\n   * @throws IOException if moving the replica from finalized directory \n   *         to rbw directory fails\n   */\n  private ReplicaInPipeline append(String bpid,\n      ReplicaInfo replicaInfo, long newGS, long estimateBlockLen)\n      throws IOException {\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n        bpid, replicaInfo.getStorageUuid(),\n        datasetSubLockStrategy.blockIdToSubLock(replicaInfo.getBlockId()))) {\n      // If the block is cached, start uncaching it.\n      if (replicaInfo.getState() != ReplicaState.FINALIZED) {\n        throw new IOException(\"Only a Finalized replica can be appended to; \"\n            + \"Replica with blk id \" + replicaInfo.getBlockId() + \" has state \"\n            + replicaInfo.getState());\n      }\n      // If the block is cached, start uncaching it.\n      cacheManager.uncacheBlock(bpid, replicaInfo.getBlockId());\n\n      // If there are any hardlinks to the block, break them.  This ensures\n      // we are not appending to a file that is part of a previous/ directory.\n      replicaInfo.breakHardLinksIfNeeded();\n\n      FsVolumeImpl v = (FsVolumeImpl)replicaInfo.getVolume();\n      ReplicaInPipeline rip = v.append(bpid, replicaInfo,\n          newGS, estimateBlockLen);\n      if (rip.getReplicaInfo().getState() != ReplicaState.RBW) {\n        throw new IOException(\"Append on block \" + replicaInfo.getBlockId() +\n            \" returned a replica of state \" + rip.getReplicaInfo().getState()\n            + \"; expected RBW\");\n      }","sourceCodeStart":1444,"sourceCodeEnd":1480,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1444-L1480","documentation":"After the public append() validated replica state, the private append(bpid, replicaInfo, newGS, estimateBlockLen) re-checks FINALIZED under the directory-level lock, immediately before the volume physically converts the finalized file back to RBW. This IOException means the replica was not FINALIZED at the moment of the actual state transition: a race changed the state between the two checks, or an internal caller bypassed the outer validation.","triggerScenarios":"Concurrent finalize/recovery changing or invalidating the replica between the outer state check (line 1419) and this inner check; direct invocation of the private append with a non-finalized ReplicaInfo.","commonSituations":"Lease recovery racing an append retry on the same block; tests or forks calling FsDatasetImpl internals directly; rare in the stock single-writer flow because the lease protocol serializes writers.","solutions":["Retry the append after the concurrent operation (lease recovery/finalize) settles; the second attempt sees the settled state","Inspect DataNode logs for interleaved operations on the same block id (append vs recover vs finalize)","If reproducible, capture jstack on the DataNode - concurrent writers on one block indicate a lease-protocol violation upstream","Upgrade to a current 3.x maintenance release; append/recovery races have been hardened across releases"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (r == null || r.getState() != ReplicaState.FINALIZED) {\n  // concurrent operation in flight; retry later instead of forcing the append\n  scheduleAppendRetry();\n  return;\n}\nfsDataset.append(b, newGS, expectedBlockLen);","typeGuard":null,"tryCatchPattern":"catch (IOException ioe) {\n  if (ioe.getMessage() != null && ioe.getMessage().startsWith(\"Only a Finalized replica\")) {\n    backoffThenRetryAppendWithFreshLeaseAndLocations(); // race with recovery/finalize\n  } else { throw ioe; }\n}","preventionTips":["Serialize writers per file: refresh the lease before every append attempt","On any append failure, refresh block locations and GS instead of replaying the identical call","Never call FsDatasetImpl internals directly; go through FsDatasetSpi append/recoverAppend which hold the proper locks"],"tags":["hdfs","datanode","append","replica-state","race-condition"],"backgroundTag":"hdfs-replica-state-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}