apache/hadoop · error · IOException

Append on block {blockId} returned a replica of state {state

Error message

Append on block {blockId} returned a replica of state {state}; expected RBW

What it means

Defensive invariant: after FsVolumeImpl.append() converts a finalized replica into a replica under construction, the result must be in RBW state before it is registered in the volumeMap. If the returned ReplicaInPipeline wraps any other state, FsDatasetImpl aborts with this IOException rather than tracking a replica in an impossible state for a write pipeline.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1477

        datasetSubLockStrategy.blockIdToSubLock(replicaInfo.getBlockId()))) {
      // If the block is cached, start uncaching it.
      if (replicaInfo.getState() != ReplicaState.FINALIZED) {
        throw new IOException("Only a Finalized replica can be appended to; "
            + "Replica with blk id " + replicaInfo.getBlockId() + " has state "
            + replicaInfo.getState());
      }
      // If the block is cached, start uncaching it.
      cacheManager.uncacheBlock(bpid, replicaInfo.getBlockId());

      // If there are any hardlinks to the block, break them.  This ensures
      // we are not appending to a file that is part of a previous/ directory.
      replicaInfo.breakHardLinksIfNeeded();

      FsVolumeImpl v = (FsVolumeImpl)replicaInfo.getVolume();
      ReplicaInPipeline rip = v.append(bpid, replicaInfo,
          newGS, estimateBlockLen);
      if (rip.getReplicaInfo().getState() != ReplicaState.RBW) {
        throw new IOException("Append on block " + replicaInfo.getBlockId() +
            " returned a replica of state " + rip.getReplicaInfo().getState()
            + "; expected RBW");
      }
      // Replace finalized replica by a RBW replica in replicas map
      volumeMap.add(bpid, rip.getReplicaInfo());
      return rip;
    }
  }

  @SuppressWarnings("serial")
  private static class MustStopExistingWriter extends Exception {
    private final ReplicaInPipeline rip;

    MustStopExistingWriter(ReplicaInPipeline rip) {
      this.rip = rip;
    }

    ReplicaInPipeline getReplicaInPipeline() {

View on GitHub (pinned to 2add963021)

Solutions

1. If a custom fsdataset factory is configured (dfs.datanode.fsdataset.factory), verify its append() returns ReplicaInPipeline with state RBW
2. Temporarily switch back to the stock FsDatasetFactory to confirm the fault is in the custom implementation
3. Collect DataNode logs plus the custom dataset source and file the bug with the vendor/project
4. On stock Hadoop, upgrade and open an HDFS JIRA with the full log line

catch (IOException ioe) {
  if (ioe.getMessage() != null && ioe.getMessage().startsWith("Append on block")) {
    // dataset/volume invariant bug: fail fast with diagnostics, do not blind-retry
    LOG.error("FsVolume.append returned non-RBW replica; dataset implementation bug", ioe);
    failOverToOtherReplicas();
  } else { throw ioe; }
}

When it happens

Trigger: FsVolumeImpl.append (or a replacement volume implementation configured via dfs.datanode.fsdataset.factory) returning a replica whose getReplicaInfo().getState() is not RBW. Stock FileFsVolume implementations never do this.

Common situations: Almost exclusively third-party or custom FsDataset/FsVolume implementations (in-memory, tiered, test fakes) misreporting state after append; on stock builds it indicates a genuine HDFS bug.

Related errors

• CreateRBW returned a replica of state {state} for block {blo
• Cannot append to a replica with unexpected generation stamp
• The new generation stamp {newGS} should be greater than the
• Cannot append to an unfinalized replica {b}
• Corrupted replica {replicaInfo} with a length of {numBytes}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fb6d6f5f1267617b. Report an issue: GitHub.