apache/hadoop · error · IOException

Only a Finalized replica can be appended to; Replica with bl

Error message

Only a Finalized replica can be appended to; Replica with blk id {blockId} has state {state}

What it means

After the public append() validated replica state, the private append(bpid, replicaInfo, newGS, estimateBlockLen) re-checks FINALIZED under the directory-level lock, immediately before the volume physically converts the finalized file back to RBW. This IOException means the replica was not FINALIZED at the moment of the actual state transition: a race changed the state between the two checks, or an internal caller bypassed the outer validation.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1462

   * bump its generation stamp to be the newGS
   * 
   * @param bpid block pool Id
   * @param replicaInfo a finalized replica
   * @param newGS new generation stamp
   * @param estimateBlockLen estimate block length
   * @return a RBW replica
   * @throws IOException if moving the replica from finalized directory 
   *         to rbw directory fails
   */
  private ReplicaInPipeline append(String bpid,
      ReplicaInfo replicaInfo, long newGS, long estimateBlockLen)
      throws IOException {
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        bpid, replicaInfo.getStorageUuid(),
        datasetSubLockStrategy.blockIdToSubLock(replicaInfo.getBlockId()))) {
      // If the block is cached, start uncaching it.
      if (replicaInfo.getState() != ReplicaState.FINALIZED) {
        throw new IOException("Only a Finalized replica can be appended to; "
            + "Replica with blk id " + replicaInfo.getBlockId() + " has state "
            + replicaInfo.getState());
      }
      // If the block is cached, start uncaching it.
      cacheManager.uncacheBlock(bpid, replicaInfo.getBlockId());

      // If there are any hardlinks to the block, break them.  This ensures
      // we are not appending to a file that is part of a previous/ directory.
      replicaInfo.breakHardLinksIfNeeded();

      FsVolumeImpl v = (FsVolumeImpl)replicaInfo.getVolume();
      ReplicaInPipeline rip = v.append(bpid, replicaInfo,
          newGS, estimateBlockLen);
      if (rip.getReplicaInfo().getState() != ReplicaState.RBW) {
        throw new IOException("Append on block " + replicaInfo.getBlockId() +
            " returned a replica of state " + rip.getReplicaInfo().getState()
            + "; expected RBW");
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the append after the concurrent operation (lease recovery/finalize) settles; the second attempt sees the settled state
2. Inspect DataNode logs for interleaved operations on the same block id (append vs recover vs finalize)
3. If reproducible, capture jstack on the DataNode - concurrent writers on one block indicate a lease-protocol violation upstream
4. Upgrade to a current 3.x maintenance release; append/recovery races have been hardened across releases

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (r == null || r.getState() != ReplicaState.FINALIZED) {
  // concurrent operation in flight; retry later instead of forcing the append
  scheduleAppendRetry();
  return;
}
fsDataset.append(b, newGS, expectedBlockLen);

catch (IOException ioe) {
  if (ioe.getMessage() != null && ioe.getMessage().startsWith("Only a Finalized replica")) {
    backoffThenRetryAppendWithFreshLeaseAndLocations(); // race with recovery/finalize
  } else { throw ioe; }
}

When it happens

Trigger: Concurrent finalize/recovery changing or invalidating the replica between the outer state check (line 1419) and this inner check; direct invocation of the private append with a non-finalized ReplicaInfo.

Common situations: Lease recovery racing an append retry on the same block; tests or forks calling FsDatasetImpl internals directly; rare in the stock single-writer flow because the lease protocol serializes writers.

Related errors

• Cannot append to an unfinalized replica {b}
• Replica is not readable, block={block}, replica={replica}
• FsDatasetSpi has not been initialized
• Failed to create temporary file for {}. File {} should be c
• Cannot append to a replica with unexpected generation stamp

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5cddc3b293db4120. Report an issue: GitHub.