{"record":{"id":"2c94ed0ed44285d4","repo":"apache/hadoop","slug":"cannot-append-to-an-unfinalized-replica-b","errorCode":null,"errorMessage":"Cannot append to an unfinalized replica {b}","messagePattern":"Cannot append to an unfinalized replica (.+?)","errorType":"exception","errorClass":"ReplicaNotFoundException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java","lineNumber":1420,"sourceCode":"      long newGS, long expectedBlockLen) throws IOException {\n    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,\n        b.getBlockPoolId(), getStorageUuidForLock(b),\n        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {\n      // If the block was successfully finalized because all packets\n      // were successfully processed at the Datanode but the ack for\n      // some of the packets were not received by the client. The client\n      // re-opens the connection and retries sending those packets.\n      // The other reason is that an \"append\" is occurring to this block.\n\n      // check the validity of the parameter\n      if (newGS < b.getGenerationStamp()) {\n        throw new IOException(\"The new generation stamp \" + newGS +\n            \" should be greater than the replica \" + b + \"'s generation stamp\");\n      }\n      ReplicaInfo replicaInfo = getReplicaInfo(b);\n      LOG.info(\"Appending to \" + replicaInfo);\n      if (replicaInfo.getState() != ReplicaState.FINALIZED) {\n        throw new ReplicaNotFoundException(\n            ReplicaNotFoundException.UNFINALIZED_REPLICA + b);\n      }\n      if (replicaInfo.getNumBytes() != expectedBlockLen) {\n        throw new IOException(\"Corrupted replica \" + replicaInfo +\n            \" with a length of \" + replicaInfo.getNumBytes() +\n            \" expected length is \" + expectedBlockLen);\n      }\n\n      FsVolumeReference ref = replicaInfo.getVolume().obtainReference();\n      ReplicaInPipeline replica = null;\n      try {\n        replica = append(b.getBlockPoolId(), replicaInfo, newGS,\n            b.getNumBytes());\n      } catch (IOException e) {\n        IOUtils.cleanupWithLogger(null, ref);\n        throw e;\n      }\n      return new ReplicaHandler(replica, ref);","sourceCodeStart":1402,"sourceCodeEnd":1438,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java#L1402-L1438","documentation":"FsDatasetImpl.append() appends only to FINALIZED replicas: the block must be closed before more bytes can be written to it. If the volumeMap entry for the block is in any other state (RBW from a live or crashed writer, TEMPORARY, or RWR after DataNode restart), it throws ReplicaNotFoundException with the UNFINALIZED_REPLICA prefix, meaning a previous writer started but never finished this block.","triggerScenarios":"A client opens a plain append pipeline (BlockReceiver.java:235, data.append) for a block whose replica on this DataNode is still RBW (writer active or crashed) or TEMPORARY/RWR. Plain append is legal only on finalized replicas; when a lease may still be outstanding the client must go through recoverAppend (BlockReceiver.java:241), which accepts FINALIZED and RBW.","commonSituations":"Appending to a file whose previous write crashed without closing; appending before the old lease is recovered; DataNode restarted mid-write leaving RWR replicas; test code calling FsDatasetSpi.append directly instead of the recovery-aware API.","solutions":["Let lease recovery run first (previous writer closed/killed, or 'hdfs debug recoverLease'), then retry; the client then uses the recoverAppend path which accepts RBW","Confirm no process still holds the file open for write: 'hdfs fsck / -openforwrite'","If the RBW replica is orphaned (no writer, no lease), NameNode block recovery will truncate/finalize it; watch DN logs for 'Recover failed append'","Persistent UNFINALIZED_REPLICA with no open lease: restart the DataNode or remove the stray replica so it is re-replicated"],"exampleFix":"// before: plain append onto a replica that may not be finalized\nReplicaHandler h = fsDataset.append(b, newGS, expectedBlockLen);\n\n// after: fall back to the recovery-aware API when the replica is not FINALIZED\nReplicaHandler h;\ntry {\n  h = fsDataset.append(b, newGS, expectedBlockLen);\n} catch (ReplicaNotFoundException e) {\n  h = fsDataset.recoverAppend(b, newGS, expectedBlockLen); // accepts FINALIZED and RBW\n}","handlingStrategy":"validation","validationCode":"Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());\nif (r == null) {\n  throw new ReplicaNotFoundException(\"non-existent replica \" + b);\n}\nif (r.getState() != ReplicaState.FINALIZED) {\n  // lease may be outstanding: use the recovery-aware API instead of plain append\n  fsDataset.recoverAppend(b, newGS, expectedBlockLen);\n} else {\n  fsDataset.append(b, newGS, expectedBlockLen);\n}","typeGuard":"boolean isAppendable(FsDatasetSpi data, ExtendedBlock b) throws IOException {\n  Replica r = data.getReplica(b.getBlockPoolId(), b.getBlockId());\n  return r != null && r.getState() == ReplicaState.FINALIZED;\n}","tryCatchPattern":"catch (ReplicaNotFoundException rnfe) {\n  if (rnfe.getMessage().contains(ReplicaNotFoundException.UNFINALIZED_REPLICA)) {\n    recoverLeaseThenRetryWithRecoverAppend();\n  } else { throw rnfe; }\n}","preventionTips":["Always close or abort DFSOutputStream so replicas are finalized before the next append","Use recoverAppend whenever a lease may still be outstanding instead of assuming a finalized replica","Monitor files open-for-write and recover stale leases before users re-open them for append"],"tags":["hdfs","datanode","append","replica-state","lease-recovery"],"backgroundTag":"hdfs-replica-state-conflict","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}