apache/hadoop · error · ReplicaNotFoundException

Replica does not exist {b}

Error message

Replica does not exist {b}

What it means

When a write pipeline is set up on a DataNode whose replica is TEMPORARY (e.g. created by a datanode-to-datanode transfer), FsDatasetImpl looks the replica up in the volumeMap before converting it to RBW. If no entry exists for the blockPoolId+blockId, ReplicaNotFoundException with the NON_EXISTENT_REPLICA prefix is thrown: the DN is asked to convert a replica it does not have.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:1819

  @Override // FsDatasetSpi
  public ReplicaInPipeline convertTemporaryToRbw(
      final ExtendedBlock b) throws IOException {
    long startTimeMs = Time.monotonicNow();
    try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.DIR,
        b.getBlockPoolId(), getStorageUuidForLock(b),
        datasetSubLockStrategy.blockIdToSubLock(b.getBlockId()))) {
      final long blockId = b.getBlockId();
      final long expectedGs = b.getGenerationStamp();
      final long visible = b.getNumBytes();
      LOG.info("Convert " + b + " from Temporary to RBW, visible length="
          + visible);

      final ReplicaInfo temp;
      {
        // get replica
        final ReplicaInfo r = volumeMap.get(b.getBlockPoolId(), blockId);
        if (r == null) {
          throw new ReplicaNotFoundException(
              ReplicaNotFoundException.NON_EXISTENT_REPLICA + b);
        }
        // check the replica's state
        if (r.getState() != ReplicaState.TEMPORARY) {
          throw new ReplicaAlreadyExistsException(
              "r.getState() != ReplicaState.TEMPORARY, r=" + r);
        }
        temp = r;
      }
      // check generation stamp
      if (temp.getGenerationStamp() != expectedGs) {
        throw new ReplicaAlreadyExistsException(
            "temp.getGenerationStamp() != expectedGs = " + expectedGs
                + ", temp=" + temp);
      }

      // TODO: check writer?
      // set writer to the current thread

View on GitHub (pinned to 2add963021)

Solutions

1. Client refreshes block locations and rebuilds the pipeline on current DNs - usually self-heals
2. Verify the DN re-registered its storage and replicas after any restart
3. If the block is missing everywhere, use 'hdfs fsck' and NN logs to decide between re-replication and loss
4. Re-open the file on the client to discard stale pipeline state

Replica r = fsDataset.getReplica(b.getBlockPoolId(), b.getBlockId());
if (r == null) {
  rebuildPipelineExcludingThisDatanode(); // wrong DN or replica vanished
  return;
}
fsDataset.convertTemporaryToRbw(b, visible);

boolean replicaExists(FsDatasetSpi data, ExtendedBlock b) throws IOException {
  return data.getReplica(b.getBlockPoolId(), b.getBlockId()) != null;
}

catch (ReplicaNotFoundException rnfe) {
  if (rnfe.getMessage().contains(ReplicaNotFoundException.NON_EXISTENT_REPLICA)) {
    pipeline = refreshLocatedBlocksAndRebuild(); // exclude this DN, retry elsewhere
  } else { throw rnfe; }
}

When it happens

Trigger: Pipeline setup targeting a DN whose temporary replica vanished from the volumeMap: the DN restarted (temp replicas are re-scanned or discarded during startup), the replica was invalidated, or the located block is stale and this DN never held the block.

Common situations: Client using stale located blocks after DN restart or topology changes; race between replica invalidation and pipeline rebuild; temp replica deleted by the directory scanner.

Related errors

• r.getState() != ReplicaState.TEMPORARY, r={r}
• temp.getGenerationStamp() != expectedGs = {expectedGs}, temp
• {numBytes} = numBytes < visible = {visible}, temp={temp}
• Failed to create temporary file for {}. File {} should not
• Failed to create temporary file for {}. File {} should be c

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/20d419ad57ccaf55. Report an issue: GitHub.