apache/hadoop · error · ReplicaNotFoundException

Cannot append to an unfinalized replica {b}

Error message

Cannot append to an unfinalized replica {b}

What it means

FsDatasetImpl.append() appends only to FINALIZED replicas: the block must be closed before more bytes can be written to it. If the volumeMap entry for the block is in any other state (RBW from a live or crashed writer, TEMPORARY, or RWR after DataNode restart), it throws ReplicaNotFoundException with the UNFINALIZED_REPLICA prefix, meaning a previous writer started but never finished this block.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1420

      long newGS, long expectedBlockLen) throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        b.getBlockPoolId(), getStorageUuidForLock(b),
        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
      // If the block was successfully finalized because all packets
      // were successfully processed at the Datanode but the ack for
      // some of the packets were not received by the client. The client
      // re-opens the connection and retries sending those packets.
      // The other reason is that an "append" is occurring to this block.

      // check the validity of the parameter
      if (newGS < b.getGenerationStamp()) {
        throw new IOException("The new generation stamp " + newGS +
            " should be greater than the replica " + b + "'s generation stamp");
      }
      ReplicaInfo replicaInfo = getReplicaInfo(b);
      LOG.info("Appending to " + replicaInfo);
      if (replicaInfo.getState() != ReplicaState.FINALIZED) {
        throw new ReplicaNotFoundException(
            ReplicaNotFoundException.UNFINALIZED_REPLICA + b);
      }
      if (replicaInfo.getNumBytes() != expectedBlockLen) {
        throw new IOException("Corrupted replica " + replicaInfo +
            " with a length of " + replicaInfo.getNumBytes() +
            " expected length is " + expectedBlockLen);
      }

      FsVolumeReference ref = replicaInfo.getVolume().obtainReference();
      ReplicaInPipeline replica = null;
      try {
        replica = append(b.getBlockPoolId(), replicaInfo, newGS,
            b.getNumBytes());
      } catch (IOException e) {
        IOUtils.cleanupWithLogger(null, ref);
        throw e;
      }
      return new ReplicaHandler(replica, ref);

View on GitHub (pinned to 2add963021)

Solutions

1. Let lease recovery run first (previous writer closed/killed, or 'hdfs debug recoverLease'), then retry; the client then uses the recoverAppend path which accepts RBW
2. Confirm no process still holds the file open for write: 'hdfs fsck / -openforwrite'
3. If the RBW replica is orphaned (no writer, no lease), NameNode block recovery will truncate/finalize it; watch DN logs for 'Recover failed append'
4. Persistent UNFINALIZED_REPLICA with no open lease: restart the DataNode or remove the stray replica so it is re-replicated

Example fix

// before: plain append onto a replica that may not be finalized
ReplicaHandler h = fsDataset.append(b, newGS, expectedBlockLen);

// after: fall back to the recovery-aware API when the replica is not FINALIZED
ReplicaHandler h;
try {
  h = fsDataset.append(b, newGS, expectedBlockLen);
} catch (ReplicaNotFoundException e) {
  h = fsDataset.recoverAppend(b, newGS, expectedBlockLen); // accepts FINALIZED and RBW
}

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (r == null) {
  throw new ReplicaNotFoundException("non-existent replica " + b);
}
if (r.getState() != ReplicaState.FINALIZED) {
  // lease may be outstanding: use the recovery-aware API instead of plain append
  fsDataset.recoverAppend(b, newGS, expectedBlockLen);
} else {
  fsDataset.append(b, newGS, expectedBlockLen);
}

boolean isAppendable(FsDatasetSpi data, ExtendedBlock b) throws IOException {
  Replica r = data.getReplica(b.getBlockPoolId(), b.getBlockId());
  return r != null && r.getState() == ReplicaState.FINALIZED;
}

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage().contains(ReplicaNotFoundException.UNFINALIZED_REPLICA)) {
    recoverLeaseThenRetryWithRecoverAppend();
  } else { throw rnfe; }
}

When it happens

Trigger: A client opens a plain append pipeline (BlockReceiver.java:235, data.append) for a block whose replica on this DataNode is still RBW (writer active or crashed) or TEMPORARY/RWR. Plain append is legal only on finalized replicas; when a lease may still be outstanding the client must go through recoverAppend (BlockReceiver.java:241), which accepts FINALIZED and RBW.

Common situations: Appending to a file whose previous write crashed without closing; appending before the old lease is recovered; DataNode restarted mid-write leaving RWR replicas; test code calling FsDatasetSpi.append directly instead of the recovery-aware API.

Related errors

• Cannot append to a replica with unexpected generation stamp
• The new generation stamp {newGS} should be greater than the
• Only a Finalized replica can be appended to; Replica with bl
• Replica {b} is not in expected state {state}
• replica.getState() != RUR, replica={replica}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2c94ed0ed44285d4. Report an issue: GitHub.